Sep 11 03:48:11.346788 ip-10-0-142-242 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 03:48:11.346801 ip-10-0-142-242 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 03:48:11.346811 ip-10-0-142-242 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 03:48:11.347136 ip-10-0-142-242 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 03:48:21.038973 ip-10-0-142-242 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 03:48:21.038987 ip-10-0-142-242 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f2628ff3c243481098fd6300fdc3698e --
Sep 11 03:50:05.877096 ip-10-0-142-242 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 03:50:06.289012 ip-10-0-142-242 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:50:06.289012 ip-10-0-142-242 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 03:50:06.289012 ip-10-0-142-242 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:50:06.289012 ip-10-0-142-242 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 03:50:06.289012 ip-10-0-142-242 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:50:06.290076 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.289943    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 03:50:06.293726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293712    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:50:06.293726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293726    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293730    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293733    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293737    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293740    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293743    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293746    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293749    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293752    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293754    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293757    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293759    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293762    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293775    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293778    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293781    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293788    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293791    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293794    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293797    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:50:06.293809 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293799    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293802    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293804    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293807    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293809    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293812    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293815    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293817    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293820    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293823    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293826    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293828    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293833    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293837    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293840    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293843    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293846    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293849    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293852    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293855    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:50:06.294288 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293858    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293861    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293864    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293866    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293869    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293871    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293874    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293877    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293880    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293882    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293885    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293887    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293890    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293892    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293895    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293898    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293901    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293904    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293907    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:50:06.294782 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293909    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293912    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293914    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293917    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293920    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293923    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293925    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293929    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293931    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293934    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293937    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293940    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293943    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293946    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293949    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293952    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293954    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293957    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293959    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293962    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:50:06.295250 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293964    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293967    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293969    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293972    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293974    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293977    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293981    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293987    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293991    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293994    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.293997    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294000    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294003    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294007    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294011    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294014    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294017    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294020    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294022    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:50:06.295758 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294704    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294710    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294713    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294716    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294719    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294722    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294725    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294728    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294731    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294733    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294736    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294738    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294741    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294744    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294746    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294749    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294751    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294754    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294757    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294760    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:50:06.296234 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294763    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294780    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294783    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294786    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294789    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294792    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294795    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294797    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294800    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294802    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294805    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294808    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294811    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294813    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294816    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294818    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294821    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294826    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294830    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:50:06.296824 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294833    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294836    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294839    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294842    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294844    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294847    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294850    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294852    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294854    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294857    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294859    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294862    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294865    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294868    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294871    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294873    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294876    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294879    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294881    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294884    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:50:06.297293 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294887    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294889    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294892    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294894    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294896    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294899    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294903    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294906    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294909    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294912    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294915    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294918    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294922    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294924    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294927    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294930    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294933    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294935    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294938    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:50:06.297784 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294940    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294943    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294945    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294948    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294950    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294953    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294957    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294960    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294962    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294965    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294967    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294970    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294974    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294977    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294980    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294982    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294985    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294987    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294990    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294992    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:50:06.298275 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.294995    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295386    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295391    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295394    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295396    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295400    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295402    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295405    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295407    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295410    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295413    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295416    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295419    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295421    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295424    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295427    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295430    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295433    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295435    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295438    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:50:06.298751 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295441    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295443    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295446    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295448    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295451    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295453    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295456    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295459    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295461    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295464    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295467    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295469    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295472    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295475    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295477    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295480    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295482    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295485    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295488    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295490    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:50:06.299251 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295493    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295495    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295498    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295500    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295503    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295506    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295509    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295511    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295514    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295517    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295520    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295522    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295525    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295528    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295530    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295532    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295535    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295538    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295540    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:50:06.299730 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295543    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295547    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295551    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295554    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295557    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295559    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295562    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295565    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295568    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295571    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295574    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295576    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295579    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295582    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295585    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295587    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295591    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295594    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295597    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:50:06.300252 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295600    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295603    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295605    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295608    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295611    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295614    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295616    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295619    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295621    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295624    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295627    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295629    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295632    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295634    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295637    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295639    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295642    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295644    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295647    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295649    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:50:06.300724 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295652    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.295656    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295728    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295787    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295795    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295799    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295810    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295816    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295824    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295828    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295832    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295835    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295838    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295841    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295844    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295847    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295850    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295853    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295857    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295860    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295864    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295867    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295871    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295874    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 03:50:06.301213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295877    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295881    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295884    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295888    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295891    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295894    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295897    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295899    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295903    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295906    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295910    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295913    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295915    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295918    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295921    2581 flags.go:64] FLAG: --enable-server="true"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295926    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295931    2581 flags.go:64] FLAG: --event-burst="100"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295934    2581 flags.go:64] FLAG: --event-qps="50"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295937    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295940    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295943    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295946    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295949    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295953    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295957    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 11 03:50:06.301809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295960    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295963    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295966    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295969    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295972    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295975    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295977    2581 flags.go:64] FLAG: --feature-gates=""
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295981    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295984    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295987    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295990    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295993    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295996    2581 flags.go:64] FLAG: --help="false"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.295999    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296002    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296005    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296008    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296011    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296014    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296017    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296020    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296023    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296027    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 03:50:06.302408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296030    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296033    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296036    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296039    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296042    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296045    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296048    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296052    2581 flags.go:64] FLAG: --lock-file=""
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296055    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296058    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296061    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296066    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296069    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296072    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296075    2581 flags.go:64] FLAG: --logging-format="text"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296078    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296081    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296084    2581 flags.go:64] FLAG: --manifest-url=""
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296087    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296091    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296094    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296098    2581 flags.go:64] FLAG: --max-pods="110"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296101    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296104    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296107    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 03:50:06.303001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296110    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296113    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296116    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296119    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296126    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296129    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296133    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296136    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296140    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296145    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296148    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296151    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296154    2581 flags.go:64] FLAG: --port="10250"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296157    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296160    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-081fd8728d05a9d32"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296164    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296167    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296170    2581 flags.go:64] FLAG: --register-node="true"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296173    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296177    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296180    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296183    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296186    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296189    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296192    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 03:50:06.303641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296195    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296198    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296201    2581 flags.go:64] FLAG: --runonce="false"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296204    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296207    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296210    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296213    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296215    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296218    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296221    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296224    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296227    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296230    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296234    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296237    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296241    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296244    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296247    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296253    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296256    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296258    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296262    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296266    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296269    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296272    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 03:50:06.304277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296275    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296278    2581 flags.go:64] FLAG: --v="2"
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296282    2581 flags.go:64] FLAG: --version="false"
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296286    2581 flags.go:64] FLAG: --vmodule=""
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296290    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296293    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296407    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296411    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296414    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296417    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296420    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296423    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296425    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296428    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296431    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296433    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296436    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296438    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296441    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296443    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296447    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:50:06.304903 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296450    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296453    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296456    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296459    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296461    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296464    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296467    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296470    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296475    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296478    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296481    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296484    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296486    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296488    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296491    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296494    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296497    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296500    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296502    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:50:06.305400 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296512    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296517    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296520    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296524    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296526    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296529    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296532    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296535    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296538    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296540    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296543    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296546    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296550    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296553    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296557    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296561    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296563    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296567    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296570    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:50:06.305886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296572    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296575    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296579    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296581    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296584    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296587    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296589    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296592    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296596    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296599    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296602    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296605    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296608    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296611    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296614    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296617    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296619    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296622    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296625    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:50:06.306370 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296627    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296630    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296632    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296635    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296638    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296640    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296644    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296647    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296649    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296652    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296655    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296657    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296660    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296663    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296665    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296669    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296672    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296674    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296677    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296679    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:50:06.306850 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296682    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296685    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296687    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296690    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296693    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296695    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.296698    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.296703    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.303738    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.303752    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.304480    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.304488    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304537    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304542    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304545    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:50:06.307373 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304549    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304552    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304555    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304558    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304561    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304563    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304566    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304569    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304571    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304574    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304576    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304579    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304582    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304585    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304587    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304590    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304593    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304595    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304598    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304601    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:50:06.307741 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304603    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304606    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304608    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304611    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304613    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304616    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304620    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304623    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304625    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304628    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304630    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304633    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304636    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304639    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304641    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304643    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304646    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304649    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304651    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304654    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:50:06.308232 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304656    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304659    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304661    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304664    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304666    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304669    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304671    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304674    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304677    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304680    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304682    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304685    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304687    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304690    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304692    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304695    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304697    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304701    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304704    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304706    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:50:06.308726 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304709    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304712    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304714    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304717    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304721    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304726    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304728    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304731    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304734    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304737    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304739    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304742    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304744    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304747    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304750    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304752    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304755    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304759    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304762    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:50:06.309231 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304777    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304781    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304785    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304788    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304791    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304794    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304797    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304799    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304802    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304805    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304808    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304810    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304813    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304816    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304819    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304821    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:50:06.309697 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304824    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.304829    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304931    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304936    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304939    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304942    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304944    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304947    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304949    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304952    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304955    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304957    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304959    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304962    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304965    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:50:06.310101 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304967    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304970    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304972    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304975    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304978    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304980    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304983    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304985    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304987    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304990    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304993    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304995    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.304998    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305000    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305003    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305006    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305008    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305011    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305014    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305016    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:50:06.310480 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305019    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305021    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305024    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305026    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305029    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305031    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305034    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305036    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305038    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305041    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305043    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305046    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305048    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305051    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305054    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305056    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305059    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305061    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305064    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305067    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:50:06.311077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305069    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305072    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305074    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305077    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305079    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305082    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305085    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305087    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305090    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305092    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305095    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305097    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305100    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305102    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305106    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305110    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305113    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305116    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305119    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305121    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:50:06.311871 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305124    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305128    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305131    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305134    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305136    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305139    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305141    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305144    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305147    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305149    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305152    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305155    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305157    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305159    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305162    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305165    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305168    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305170    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305174    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:50:06.312355 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305177    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:50:06.312836 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305180    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:50:06.312836 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305183    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:50:06.312836 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305186    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:50:06.312836 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305188    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:50:06.312836 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305191    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:50:06.312836 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:06.305193    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:50:06.312836 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.305198    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:50:06.312836 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.305288    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 03:50:06.312836 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.307192    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 03:50:06.312836 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.308071    2581 server.go:1019] "Starting client certificate rotation"
Sep 11 03:50:06.312836 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.308166    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 03:50:06.312836 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.308836    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 03:50:06.312836 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.310514    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 03:50:06.317807 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.317782    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 03:50:06.329803 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.329787    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 11 03:50:06.333095 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.333081    2581 log.go:25] "Validated CRI v1 image API"
Sep 11 03:50:06.334012 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.333996    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 03:50:06.337297 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.337280    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 03:50:06.340091 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.340072    2581 fs.go:135] Filesystem UUIDs: map[54bb5e24-7f34-46fb-9403-63a9c7633c79:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bdf4a627-d003-43cf-b22a-0e2eb536739f:/dev/nvme0n1p3]
Sep 11 03:50:06.340140 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.340090    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 03:50:06.345859 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.345731    2581 manager.go:216] Machine: {Timestamp:2026-09-11 03:50:06.344375912 +0000 UTC m=+0.352021561 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101923 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec276ae5da43a2ad2426ad44107041ae SystemUUID:ec276ae5-da43-a2ad-2426-ad44107041ae BootID:f2628ff3-c243-4810-98fd-6300fdc3698e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:10:01:56:92:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:10:01:56:92:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:bf:04:31:9b:23 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 03:50:06.345859 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.345855    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 03:50:06.345964 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.345937    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 03:50:06.349102 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.349074    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 03:50:06.349271 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.349106    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-242.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 03:50:06.349884 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.349873    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 03:50:06.349936 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.349889    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 03:50:06.349936 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.349900    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 03:50:06.349936 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.349920    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 03:50:06.350922 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.350910    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 03:50:06.355350 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.355337    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 03:50:06.355473 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.355464    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 03:50:06.355513 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.355493    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 03:50:06.355513 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.355510    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 03:50:06.355565 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.355528    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 03:50:06.355565 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.355541    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 11 03:50:06.355565 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.355555    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 03:50:06.355932 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.355921    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 03:50:06.355974 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.355945    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 03:50:06.358542 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.358526    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 03:50:06.361544 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.361527    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 03:50:06.361632 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.361576    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 03:50:06.364757 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.364737    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 03:50:06.364833 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.364782    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 03:50:06.364833 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.364794    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 03:50:06.364833 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.364803    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 03:50:06.364833 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.364816    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 03:50:06.364833 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.364825    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 03:50:06.364977 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.364844    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 03:50:06.364977 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.364854    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 03:50:06.364977 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.364864    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 03:50:06.364977 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.364872    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 03:50:06.364977 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.364884    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 03:50:06.364977 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.364892    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 03:50:06.364977 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.364918    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 03:50:06.364977 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.364926    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 03:50:06.365635 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.365613    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 03:50:06.365696 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.365619    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 03:50:06.368806 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.368792    2581 server.go:1268] "Started kubelet"
Sep 11 03:50:06.368911 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.368860    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 03:50:06.368963 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.368897    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 03:50:06.369013 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.368965    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 03:50:06.369660 ip-10-0-142-242 systemd[1]: Started Kubernetes Kubelet.
Sep 11 03:50:06.370869 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.370847    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nvlkc"
Sep 11 03:50:06.370956 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.370875    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 03:50:06.371192 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.371113    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 03:50:06.373376 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.373358    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 03:50:06.374758 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.374741    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-242.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 03:50:06.375132 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.375112    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 03:50:06.375807 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.374723    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-242.ec2.internal.18d427b414c164ef  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-242.ec2.internal,UID:ip-10-0-142-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-242.ec2.internal,},FirstTimestamp:2026-09-11 03:50:06.368744687 +0000 UTC m=+0.376390332,LastTimestamp:2026-09-11 03:50:06.368744687 +0000 UTC m=+0.376390332,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-242.ec2.internal,}"
Sep 11 03:50:06.375897 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.375827    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 03:50:06.375957 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.375909    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 11 03:50:06.376424 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.376405    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 03:50:06.376424 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.376409    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 03:50:06.376535 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.376435    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 03:50:06.376535 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.376517    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 03:50:06.376535 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.376527    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 03:50:06.377028 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.377010    2581 factory.go:153] Registering CRI-O factory
Sep 11 03:50:06.377108 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.377048    2581 factory.go:223] Registration of the crio container factory successfully
Sep 11 03:50:06.377108 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.377105    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 03:50:06.377173 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.377114    2581 factory.go:55] Registering systemd factory
Sep 11 03:50:06.377173 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.377121    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 11 03:50:06.377173 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.377141    2581 factory.go:103] Registering Raw factory
Sep 11 03:50:06.377173 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.377153    2581 manager.go:1196] Started watching for new ooms in manager
Sep 11 03:50:06.377568 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.377541    2581 manager.go:349] Starting recovery of all containers
Sep 11 03:50:06.379214 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.379190    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nvlkc"
Sep 11 03:50:06.381422 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.381396    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 03:50:06.381960 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.381927    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 03:50:06.384191 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.384157    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 03:50:06.388365 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.388347    2581 manager.go:354] Recovery completed
Sep 11 03:50:06.393618 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.393605    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:50:06.395592 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.395575    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:50:06.395678 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.395606    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:50:06.395678 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.395621    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:50:06.396292 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.396279    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 03:50:06.396292 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.396291    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 03:50:06.396413 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.396305    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 03:50:06.396413 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.396326    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:50:06.398183 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.397159    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:50:06.398183 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.397190    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:50:06.398183 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.397202    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:50:06.398494 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.398474    2581 policy_none.go:49] "None policy: Start"
Sep 11 03:50:06.398566 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.398500    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 03:50:06.398566 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.398515    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 03:50:06.400442 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.400429    2581 policy_none.go:47] "Start"
Sep 11 03:50:06.427340 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.427326    2581 manager.go:337] "Starting Device Plugin manager"
Sep 11 03:50:06.448527 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.427362    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 03:50:06.448527 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.427372    2581 server.go:87] "Starting device plugin registration server"
Sep 11 03:50:06.448527 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.427609    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 03:50:06.448527 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.427624    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 03:50:06.448527 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.427676    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 03:50:06.448527 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.427749    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 03:50:06.448527 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.427758    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 03:50:06.448527 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.428391    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 03:50:06.448527 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.428564    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 11 03:50:06.469416 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.469396    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 03:50:06.469529 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.469455    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 03:50:06.469529 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.469483    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 03:50:06.469623 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.469572    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 03:50:06.471747 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.471733    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:50:06.528743 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.528713    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:50:06.529662 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.529646    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:50:06.529750 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.529675    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:50:06.529750 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.529687    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:50:06.529750 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.529716    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.535465 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.535448    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.535551 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.535475    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-242.ec2.internal\": node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 11 03:50:06.551280 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.551224    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 11 03:50:06.570312 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.570279    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal"]
Sep 11 03:50:06.570385 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.570358    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:50:06.571463 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.571443    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:50:06.571528 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.571476    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:50:06.571528 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.571486    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:50:06.573646 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.573633    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:50:06.573790 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.573757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.573847 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.573809    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:50:06.575860 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.575844    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:50:06.575994 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.575876    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:50:06.575994 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.575889    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:50:06.576303 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.576282    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:50:06.576385 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.576318    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:50:06.576385 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.576332    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:50:06.577888 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.577874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/79d681e26c169b5ac07eacf2cb3d45e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal\" (UID: \"79d681e26c169b5ac07eacf2cb3d45e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.577958 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.577896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79d681e26c169b5ac07eacf2cb3d45e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal\" (UID: \"79d681e26c169b5ac07eacf2cb3d45e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.577958 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.577912    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/427abdc7e2784c3c5f6bbe0ee342a5b6-config\") pod \"kube-apiserver-proxy-ip-10-0-142-242.ec2.internal\" (UID: \"427abdc7e2784c3c5f6bbe0ee342a5b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.578383 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.578367    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.578418 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.578396    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:50:06.579114 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.579097    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:50:06.579177 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.579129    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:50:06.579177 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.579144    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:50:06.601374 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.601350    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-242.ec2.internal\" not found" node="ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.605616 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.605601    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-242.ec2.internal\" not found" node="ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.651603 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.651579    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 11 03:50:06.678368 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.678345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/79d681e26c169b5ac07eacf2cb3d45e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal\" (UID: \"79d681e26c169b5ac07eacf2cb3d45e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.678456 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.678375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79d681e26c169b5ac07eacf2cb3d45e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal\" (UID: \"79d681e26c169b5ac07eacf2cb3d45e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.678456 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.678393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/427abdc7e2784c3c5f6bbe0ee342a5b6-config\") pod \"kube-apiserver-proxy-ip-10-0-142-242.ec2.internal\" (UID: \"427abdc7e2784c3c5f6bbe0ee342a5b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.678456 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.678420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/427abdc7e2784c3c5f6bbe0ee342a5b6-config\") pod \"kube-apiserver-proxy-ip-10-0-142-242.ec2.internal\" (UID: \"427abdc7e2784c3c5f6bbe0ee342a5b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.678456 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.678442    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79d681e26c169b5ac07eacf2cb3d45e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal\" (UID: \"79d681e26c169b5ac07eacf2cb3d45e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.678582 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.678458    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/79d681e26c169b5ac07eacf2cb3d45e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal\" (UID: \"79d681e26c169b5ac07eacf2cb3d45e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.751845 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.751808    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 11 03:50:06.852515 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.852412    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 11 03:50:06.904002 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.903980    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.908925 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:06.908674    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:06.952633 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:06.952606    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 11 03:50:08.438333 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:08.438312    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 11 03:50:08.538882 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:08.538810    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 11 03:50:08.639445 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:08.639413    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 11 03:50:08.692937 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:08.692912    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 03:50:08.693547 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:08.693099    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 03:50:08.740257 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:08.740219    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 11 03:50:08.769779 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:08.769727    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 03:45:07 +0000 UTC" deadline="2028-03-16 17:33:37.338338314 +0000 UTC"
Sep 11 03:50:08.769779 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:08.769770    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13261h43m28.568572983s"
Sep 11 03:50:08.775440 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:08.775361    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:50:08.776303 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:08.776282    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:50:08.861457 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:08.861397    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:08.869483 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:08.869462    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 03:50:08.870469 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:08.870456    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 11 03:50:08.879468 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:08.879455    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 03:50:08.889086 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:08.889071    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:50:08.927942 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:08.927912    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427abdc7e2784c3c5f6bbe0ee342a5b6.slice/crio-1659171747dd59e5c03126586a71a5913cf4a31a14fe1aaf6c6cf600848128af WatchSource:0}: Error finding container 1659171747dd59e5c03126586a71a5913cf4a31a14fe1aaf6c6cf600848128af: Status 404 returned error can't find the container with id 1659171747dd59e5c03126586a71a5913cf4a31a14fe1aaf6c6cf600848128af
Sep 11 03:50:08.928435 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:08.928417    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d681e26c169b5ac07eacf2cb3d45e4.slice/crio-1e8a6a173ffd716fc19c646da0adc2b9299c8b0e617d76f5e9bdbbcc3b9be428 WatchSource:0}: Error finding container 1e8a6a173ffd716fc19c646da0adc2b9299c8b0e617d76f5e9bdbbcc3b9be428: Status 404 returned error can't find the container with id 1e8a6a173ffd716fc19c646da0adc2b9299c8b0e617d76f5e9bdbbcc3b9be428
Sep 11 03:50:08.932639 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:08.932622    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 03:50:09.742256 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.742226    2581 apiserver.go:52] "Watching apiserver"
Sep 11 03:50:09.751372 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.751345    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 03:50:09.751810 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.751777    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-2qtbp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2","openshift-cluster-node-tuning-operator/tuned-kmrwl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal","openshift-multus/network-metrics-daemon-6rpfl","openshift-network-diagnostics/network-check-target-t7sgk","openshift-network-operator/iptables-alerter-dpprn","openshift-ovn-kubernetes/ovnkube-node-bxqls","kube-system/global-pull-secret-syncer-6bs2d","kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal","openshift-dns/node-resolver-4cm6g","openshift-image-registry/node-ca-mkxx2","openshift-multus/multus-additional-cni-plugins-tvdds","openshift-multus/multus-vqrfz"]
Sep 11 03:50:09.754330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.754308    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dpprn"
Sep 11 03:50:09.756635 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.756496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.757141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.756932    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fdgwq\""
Sep 11 03:50:09.757141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.756997    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:50:09.757141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.757010    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 03:50:09.757141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.757078    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 03:50:09.758715 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.758691    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.758812 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.758769    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 03:50:09.758877 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.758830    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 03:50:09.759218 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.759189    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 03:50:09.759328 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.759239    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p9xjd\""
Sep 11 03:50:09.760587 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.760569    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 03:50:09.760825 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.760807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.761036 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.761005    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:50:09.761437 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.761416    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 03:50:09.761529 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.761501    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gc9rn\""
Sep 11 03:50:09.763942 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.763915    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.765151 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.765136    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 03:50:09.765243 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.765209    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 03:50:09.765508 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.765489    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 03:50:09.765612 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.765508    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lxgqb\""
Sep 11 03:50:09.765709 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.765692    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 03:50:09.767554 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.767526    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 03:50:09.767675 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.767618    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 03:50:09.767738 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.767701    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v4qq8\""
Sep 11 03:50:09.769240 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.769221    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4cm6g"
Sep 11 03:50:09.772332 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.771717    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 03:50:09.772332 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.771942    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-75kvp\""
Sep 11 03:50:09.772332 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.772156    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 03:50:09.772604 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.772397    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 03:50:09.772662 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.772638    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.773389 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-system-cni-dir\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.773389 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773065    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-run-netns\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.773389 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-multus-conf-dir\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.773389 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-run-multus-certs\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.773389 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5ad581b-7f8b-4747-8aad-7b59824edac5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.773389 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5ad581b-7f8b-4747-8aad-7b59824edac5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.773389 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-sysconfig\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.773389 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-sysctl-conf\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.773389 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-run\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.773389 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/443eee6f-26ee-439c-bd85-2bc0b92fdd14-multus-daemon-config\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.773389 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-socket-dir\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.773389 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95wvd\" (UniqueName: \"kubernetes.io/projected/802d5da6-9490-4950-8c1b-da0454e85ad8-kube-api-access-95wvd\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-host\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-multus-socket-dir-parent\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-var-lib-cni-bin\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-etc-kubernetes\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-registration-dir\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-modprobe-d\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-sys\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-var-lib-kubelet\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/443eee6f-26ee-439c-bd85-2bc0b92fdd14-cni-binary-copy\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773686    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-var-lib-cni-multus\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773708    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-var-lib-kubelet\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773731    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5ad581b-7f8b-4747-8aad-7b59824edac5-cni-binary-copy\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773753    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-sys-fs\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-systemd\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-cnibin\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.774174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-run-k8s-cni-cncf-io\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8s8z\" (UniqueName: \"kubernetes.io/projected/f5ad581b-7f8b-4747-8aad-7b59824edac5-kube-api-access-s8s8z\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/673f4848-26b9-4885-8857-b0cb058adb40-host-slash\") pod \"iptables-alerter-dpprn\" (UID: \"673f4848-26b9-4885-8857-b0cb058adb40\") " pod="openshift-network-operator/iptables-alerter-dpprn"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-device-dir\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-sysctl-d\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-hostroot\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqd5\" (UniqueName: \"kubernetes.io/projected/443eee6f-26ee-439c-bd85-2bc0b92fdd14-kube-api-access-nrqd5\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.773995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ad581b-7f8b-4747-8aad-7b59824edac5-system-cni-dir\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.774043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5ad581b-7f8b-4747-8aad-7b59824edac5-cnibin\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.774065    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/673f4848-26b9-4885-8857-b0cb058adb40-iptables-alerter-script\") pod \"iptables-alerter-dpprn\" (UID: \"673f4848-26b9-4885-8857-b0cb058adb40\") " pod="openshift-network-operator/iptables-alerter-dpprn"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.774090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-kubernetes\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.774128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-lib-modules\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.774236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-multus-cni-dir\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.774263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sdfd\" (UniqueName: \"kubernetes.io/projected/673f4848-26b9-4885-8857-b0cb058adb40-kube-api-access-8sdfd\") pod \"iptables-alerter-dpprn\" (UID: \"673f4848-26b9-4885-8857-b0cb058adb40\") " pod="openshift-network-operator/iptables-alerter-dpprn"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.774287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-etc-selinux\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.774311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-tuned\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.775161 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.774342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-tmp\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.776077 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.774374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55tgt\" (UniqueName: \"kubernetes.io/projected/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-kube-api-access-55tgt\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.776077 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.774403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5ad581b-7f8b-4747-8aad-7b59824edac5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.776077 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.774493    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:09.776077 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:09.774571    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:09.776077 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.775373    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 03:50:09.776077 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.775411    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 03:50:09.776077 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.775645    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 03:50:09.776911 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.776550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:09.776911 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:09.776622    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:09.776911 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.776747    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 03:50:09.776911 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.776767    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-l77nx\""
Sep 11 03:50:09.777171 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.776931    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 03:50:09.777171 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.776948    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 03:50:09.779406 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.779344    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mkxx2"
Sep 11 03:50:09.781823 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.781554    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:09.781823 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:09.781615    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:09.781823 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.781714    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 03:50:09.781823 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.781731    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 03:50:09.782065 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.781939    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ghklm\""
Sep 11 03:50:09.782065 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.781986    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 03:50:09.784293 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.784077    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2qtbp"
Sep 11 03:50:09.786550 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.786418    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gqkq8\""
Sep 11 03:50:09.786550 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.786445    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 03:50:09.786724 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.786641    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 03:50:09.792039 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.791949    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zfpwj"
Sep 11 03:50:09.799179 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.799159    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zfpwj"
Sep 11 03:50:09.819284 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.819261    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:50:09.858764 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.858717    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal" event={"ID":"79d681e26c169b5ac07eacf2cb3d45e4","Type":"ContainerStarted","Data":"1e8a6a173ffd716fc19c646da0adc2b9299c8b0e617d76f5e9bdbbcc3b9be428"}
Sep 11 03:50:09.860155 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.860130    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal" event={"ID":"427abdc7e2784c3c5f6bbe0ee342a5b6","Type":"ContainerStarted","Data":"1659171747dd59e5c03126586a71a5913cf4a31a14fe1aaf6c6cf600848128af"}
Sep 11 03:50:09.862890 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.862872    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 03:50:09.874890 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.874866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5ad581b-7f8b-4747-8aad-7b59824edac5-cnibin\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.874997 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.874898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/673f4848-26b9-4885-8857-b0cb058adb40-iptables-alerter-script\") pod \"iptables-alerter-dpprn\" (UID: \"673f4848-26b9-4885-8857-b0cb058adb40\") " pod="openshift-network-operator/iptables-alerter-dpprn"
Sep 11 03:50:09.874997 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.874916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-lib-modules\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.874997 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.874933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-multus-cni-dir\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.875173 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.874989    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5ad581b-7f8b-4747-8aad-7b59824edac5-cnibin\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.875173 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-multus-cni-dir\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.875173 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875051    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-kubelet\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.875173 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-ovnkube-script-lib\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.875173 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-etc-selinux\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.875173 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-lib-modules\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.875173 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-tuned\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.875173 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55tgt\" (UniqueName: \"kubernetes.io/projected/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-kube-api-access-55tgt\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-etc-selinux\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5ad581b-7f8b-4747-8aad-7b59824edac5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-system-cni-dir\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-multus-conf-dir\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-run-multus-certs\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5ad581b-7f8b-4747-8aad-7b59824edac5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-run-ovn\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-sysconfig\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-sysctl-conf\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-run\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-run-netns\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-host\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-multus-socket-dir-parent\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875534    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-var-lib-cni-bin\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-sysconfig\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.875570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-ovn-node-metrics-cert\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-modprobe-d\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-host\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-sysctl-conf\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-sys\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875483    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-system-cni-dir\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875635    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-run\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/443eee6f-26ee-439c-bd85-2bc0b92fdd14-cni-binary-copy\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-var-lib-kubelet\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-multus-socket-dir-parent\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5ad581b-7f8b-4747-8aad-7b59824edac5-cni-binary-copy\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-multus-conf-dir\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-etc-openvswitch\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-ovnkube-config\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-systemd\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875834    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-modprobe-d\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-sys\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.876573 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-cnibin\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-var-lib-kubelet\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-run-k8s-cni-cncf-io\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-run-k8s-cni-cncf-io\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8s8z\" (UniqueName: \"kubernetes.io/projected/f5ad581b-7f8b-4747-8aad-7b59824edac5-kube-api-access-s8s8z\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-env-overrides\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.875985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ca873b5-3619-4087-a38d-221452120a26-agent-certs\") pod \"konnectivity-agent-2qtbp\" (UID: \"0ca873b5-3619-4087-a38d-221452120a26\") " pod="kube-system/konnectivity-agent-2qtbp"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-device-dir\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-sysctl-d\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-hostroot\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqd5\" (UniqueName: \"kubernetes.io/projected/443eee6f-26ee-439c-bd85-2bc0b92fdd14-kube-api-access-nrqd5\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876109    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ad581b-7f8b-4747-8aad-7b59824edac5-system-cni-dir\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876141    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-run-ovn-kubernetes\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/673f4848-26b9-4885-8857-b0cb058adb40-iptables-alerter-script\") pod \"iptables-alerter-dpprn\" (UID: \"673f4848-26b9-4885-8857-b0cb058adb40\") " pod="openshift-network-operator/iptables-alerter-dpprn"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hwcb\" (UniqueName: \"kubernetes.io/projected/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-kube-api-access-4hwcb\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-kubernetes\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.877453 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-kubernetes\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sdfd\" (UniqueName: \"kubernetes.io/projected/673f4848-26b9-4885-8857-b0cb058adb40-kube-api-access-8sdfd\") pod \"iptables-alerter-dpprn\" (UID: \"673f4848-26b9-4885-8857-b0cb058adb40\") " pod="openshift-network-operator/iptables-alerter-dpprn"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-node-log\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-systemd\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-cni-netd\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-cnibin\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-kubelet-config\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j77rq\" (UniqueName: \"kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq\") pod \"network-check-target-t7sgk\" (UID: \"7c125f86-ef79-4811-ab32-abacb2b62402\") " pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876388    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-run-multus-certs\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-tmp\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x652m\" (UniqueName: \"kubernetes.io/projected/657845f4-5ec8-4c63-9d6a-f9ca4273be0e-kube-api-access-x652m\") pod \"node-ca-mkxx2\" (UID: \"657845f4-5ec8-4c63-9d6a-f9ca4273be0e\") " pod="openshift-image-registry/node-ca-mkxx2"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-sysctl-d\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs\") pod \"network-metrics-daemon-6rpfl\" (UID: \"af6ebd1b-3025-4a6f-9822-aa6606b8fa28\") " pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876476    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-hostroot\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-run-netns\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5ad581b-7f8b-4747-8aad-7b59824edac5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-run-openvswitch\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.878330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ca873b5-3619-4087-a38d-221452120a26-konnectivity-ca\") pod \"konnectivity-agent-2qtbp\" (UID: \"0ca873b5-3619-4087-a38d-221452120a26\") " pod="kube-system/konnectivity-agent-2qtbp"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/443eee6f-26ee-439c-bd85-2bc0b92fdd14-multus-daemon-config\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-systemd-units\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ad581b-7f8b-4747-8aad-7b59824edac5-system-cni-dir\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-var-lib-openvswitch\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/443eee6f-26ee-439c-bd85-2bc0b92fdd14-cni-binary-copy\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5ad581b-7f8b-4747-8aad-7b59824edac5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-var-lib-cni-bin\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876686    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-dbus\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2dl\" (UniqueName: \"kubernetes.io/projected/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-kube-api-access-kp2dl\") pod \"network-metrics-daemon-6rpfl\" (UID: \"af6ebd1b-3025-4a6f-9822-aa6606b8fa28\") " pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876781    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-run-netns\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5ad581b-7f8b-4747-8aad-7b59824edac5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5ad581b-7f8b-4747-8aad-7b59824edac5-cni-binary-copy\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-device-dir\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-socket-dir\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876908    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-socket-dir\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95wvd\" (UniqueName: \"kubernetes.io/projected/802d5da6-9490-4950-8c1b-da0454e85ad8-kube-api-access-95wvd\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.879488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-etc-kubernetes\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.876982    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-log-socket\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/657845f4-5ec8-4c63-9d6a-f9ca4273be0e-serviceca\") pod \"node-ca-mkxx2\" (UID: \"657845f4-5ec8-4c63-9d6a-f9ca4273be0e\") " pod="openshift-image-registry/node-ca-mkxx2"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-run-systemd\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-registration-dir\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877086    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-etc-kubernetes\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-var-lib-kubelet\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-var-lib-cni-multus\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/673f4848-26b9-4885-8857-b0cb058adb40-host-slash\") pod \"iptables-alerter-dpprn\" (UID: \"673f4848-26b9-4885-8857-b0cb058adb40\") " pod="openshift-network-operator/iptables-alerter-dpprn"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-slash\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5ad581b-7f8b-4747-8aad-7b59824edac5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-var-lib-kubelet\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877299    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5604fe2c-0315-490c-a234-cae850ae2d55-hosts-file\") pod \"node-resolver-4cm6g\" (UID: \"5604fe2c-0315-490c-a234-cae850ae2d55\") " pod="openshift-dns/node-resolver-4cm6g"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-registration-dir\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/673f4848-26b9-4885-8857-b0cb058adb40-host-slash\") pod \"iptables-alerter-dpprn\" (UID: \"673f4848-26b9-4885-8857-b0cb058adb40\") " pod="openshift-network-operator/iptables-alerter-dpprn"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/443eee6f-26ee-439c-bd85-2bc0b92fdd14-host-var-lib-cni-multus\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5604fe2c-0315-490c-a234-cae850ae2d55-tmp-dir\") pod \"node-resolver-4cm6g\" (UID: \"5604fe2c-0315-490c-a234-cae850ae2d55\") " pod="openshift-dns/node-resolver-4cm6g"
Sep 11 03:50:09.880338 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.881141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-sys-fs\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.881141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/443eee6f-26ee-439c-bd85-2bc0b92fdd14-multus-daemon-config\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.881141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877449    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.881141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877462    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/802d5da6-9490-4950-8c1b-da0454e85ad8-sys-fs\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.881141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-cni-bin\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.881141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jphws\" (UniqueName: \"kubernetes.io/projected/5604fe2c-0315-490c-a234-cae850ae2d55-kube-api-access-jphws\") pod \"node-resolver-4cm6g\" (UID: \"5604fe2c-0315-490c-a234-cae850ae2d55\") " pod="openshift-dns/node-resolver-4cm6g"
Sep 11 03:50:09.881141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.877536    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/657845f4-5ec8-4c63-9d6a-f9ca4273be0e-host\") pod \"node-ca-mkxx2\" (UID: \"657845f4-5ec8-4c63-9d6a-f9ca4273be0e\") " pod="openshift-image-registry/node-ca-mkxx2"
Sep 11 03:50:09.881141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.879380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-etc-tuned\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.881141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.879433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-tmp\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.883471 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.883446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55tgt\" (UniqueName: \"kubernetes.io/projected/9d2d0186-ceb5-4b85-b5d1-3a6687fb8799-kube-api-access-55tgt\") pod \"tuned-kmrwl\" (UID: \"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799\") " pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:09.884097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.883795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8s8z\" (UniqueName: \"kubernetes.io/projected/f5ad581b-7f8b-4747-8aad-7b59824edac5-kube-api-access-s8s8z\") pod \"multus-additional-cni-plugins-tvdds\" (UID: \"f5ad581b-7f8b-4747-8aad-7b59824edac5\") " pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:09.884097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.883798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqd5\" (UniqueName: \"kubernetes.io/projected/443eee6f-26ee-439c-bd85-2bc0b92fdd14-kube-api-access-nrqd5\") pod \"multus-vqrfz\" (UID: \"443eee6f-26ee-439c-bd85-2bc0b92fdd14\") " pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:09.884244 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.884143    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sdfd\" (UniqueName: \"kubernetes.io/projected/673f4848-26b9-4885-8857-b0cb058adb40-kube-api-access-8sdfd\") pod \"iptables-alerter-dpprn\" (UID: \"673f4848-26b9-4885-8857-b0cb058adb40\") " pod="openshift-network-operator/iptables-alerter-dpprn"
Sep 11 03:50:09.884902 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.884882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95wvd\" (UniqueName: \"kubernetes.io/projected/802d5da6-9490-4950-8c1b-da0454e85ad8-kube-api-access-95wvd\") pod \"aws-ebs-csi-driver-node-sb9r2\" (UID: \"802d5da6-9490-4950-8c1b-da0454e85ad8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:09.978514 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-systemd-units\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.978680 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-var-lib-openvswitch\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.978680 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-dbus\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:09.978680 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978572    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2dl\" (UniqueName: \"kubernetes.io/projected/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-kube-api-access-kp2dl\") pod \"network-metrics-daemon-6rpfl\" (UID: \"af6ebd1b-3025-4a6f-9822-aa6606b8fa28\") " pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:09.978680 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-log-socket\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.978680 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978604    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-systemd-units\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.978680 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/657845f4-5ec8-4c63-9d6a-f9ca4273be0e-serviceca\") pod \"node-ca-mkxx2\" (UID: \"657845f4-5ec8-4c63-9d6a-f9ca4273be0e\") " pod="openshift-image-registry/node-ca-mkxx2"
Sep 11 03:50:09.978680 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-run-systemd\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.978680 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-slash\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5604fe2c-0315-490c-a234-cae850ae2d55-hosts-file\") pod \"node-resolver-4cm6g\" (UID: \"5604fe2c-0315-490c-a234-cae850ae2d55\") " pod="openshift-dns/node-resolver-4cm6g"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-var-lib-openvswitch\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5604fe2c-0315-490c-a234-cae850ae2d55-tmp-dir\") pod \"node-resolver-4cm6g\" (UID: \"5604fe2c-0315-490c-a234-cae850ae2d55\") " pod="openshift-dns/node-resolver-4cm6g"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-cni-bin\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-dbus\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jphws\" (UniqueName: \"kubernetes.io/projected/5604fe2c-0315-490c-a234-cae850ae2d55-kube-api-access-jphws\") pod \"node-resolver-4cm6g\" (UID: \"5604fe2c-0315-490c-a234-cae850ae2d55\") " pod="openshift-dns/node-resolver-4cm6g"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978812    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/657845f4-5ec8-4c63-9d6a-f9ca4273be0e-host\") pod \"node-ca-mkxx2\" (UID: \"657845f4-5ec8-4c63-9d6a-f9ca4273be0e\") " pod="openshift-image-registry/node-ca-mkxx2"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-run-systemd\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-kubelet\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-ovnkube-script-lib\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978891    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978894    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-slash\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-run-ovn\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5604fe2c-0315-490c-a234-cae850ae2d55-hosts-file\") pod \"node-resolver-4cm6g\" (UID: \"5604fe2c-0315-490c-a234-cae850ae2d55\") " pod="openshift-dns/node-resolver-4cm6g"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-log-socket\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.978969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-run-ovn\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.979011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-cni-bin\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.979097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.979038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5604fe2c-0315-490c-a234-cae850ae2d55-tmp-dir\") pod \"node-resolver-4cm6g\" (UID: \"5604fe2c-0315-490c-a234-cae850ae2d55\") " pod="openshift-dns/node-resolver-4cm6g"
Sep 11 03:50:09.979871 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:09.979044    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:09.979871 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.979078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/657845f4-5ec8-4c63-9d6a-f9ca4273be0e-host\") pod \"node-ca-mkxx2\" (UID: \"657845f4-5ec8-4c63-9d6a-f9ca4273be0e\") " pod="openshift-image-registry/node-ca-mkxx2"
Sep 11 03:50:09.979871 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.979105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-run-netns\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.979871 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:09.979480    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret podName:69e1dbc6-8f47-4733-9be1-d76b49f7f4d1 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:10.479450449 +0000 UTC m=+3.101843459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret") pod "global-pull-secret-syncer-6bs2d" (UID: "69e1dbc6-8f47-4733-9be1-d76b49f7f4d1") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:09.979871 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.979523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-run-netns\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.979871 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.979544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-ovn-node-metrics-cert\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.979871 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.979811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/657845f4-5ec8-4c63-9d6a-f9ca4273be0e-serviceca\") pod \"node-ca-mkxx2\" (UID: \"657845f4-5ec8-4c63-9d6a-f9ca4273be0e\") " pod="openshift-image-registry/node-ca-mkxx2"
Sep 11 03:50:09.979871 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.979829    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-kubelet\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.980238 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.979976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-etc-openvswitch\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.980238 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-ovnkube-config\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.980238 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980091    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-etc-openvswitch\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.980238 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-ovnkube-script-lib\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.980238 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-env-overrides\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.980238 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ca873b5-3619-4087-a38d-221452120a26-agent-certs\") pod \"konnectivity-agent-2qtbp\" (UID: \"0ca873b5-3619-4087-a38d-221452120a26\") " pod="kube-system/konnectivity-agent-2qtbp"
Sep 11 03:50:09.980238 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-run-ovn-kubernetes\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.980526 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.980526 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hwcb\" (UniqueName: \"kubernetes.io/projected/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-kube-api-access-4hwcb\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.980526 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-node-log\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.980526 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-cni-netd\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.980526 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-kubelet-config\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:09.980526 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77rq\" (UniqueName: \"kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq\") pod \"network-check-target-t7sgk\" (UID: \"7c125f86-ef79-4811-ab32-abacb2b62402\") " pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:09.980526 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x652m\" (UniqueName: \"kubernetes.io/projected/657845f4-5ec8-4c63-9d6a-f9ca4273be0e-kube-api-access-x652m\") pod \"node-ca-mkxx2\" (UID: \"657845f4-5ec8-4c63-9d6a-f9ca4273be0e\") " pod="openshift-image-registry/node-ca-mkxx2"
Sep 11 03:50:09.980526 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs\") pod \"network-metrics-daemon-6rpfl\" (UID: \"af6ebd1b-3025-4a6f-9822-aa6606b8fa28\") " pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:09.980526 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-run-openvswitch\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.980899 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ca873b5-3619-4087-a38d-221452120a26-konnectivity-ca\") pod \"konnectivity-agent-2qtbp\" (UID: \"0ca873b5-3619-4087-a38d-221452120a26\") " pod="kube-system/konnectivity-agent-2qtbp"
Sep 11 03:50:09.980899 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-ovnkube-config\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.981124 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.981098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ca873b5-3619-4087-a38d-221452120a26-konnectivity-ca\") pod \"konnectivity-agent-2qtbp\" (UID: \"0ca873b5-3619-4087-a38d-221452120a26\") " pod="kube-system/konnectivity-agent-2qtbp"
Sep 11 03:50:09.981204 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.981189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-node-log\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.981280 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.981253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-cni-netd\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.981346 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.981325    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-kubelet-config\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:09.981498 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.980985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-env-overrides\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.981928 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:09.981867    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:09.982057 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:09.981940    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs podName:af6ebd1b-3025-4a6f-9822-aa6606b8fa28 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:10.481923764 +0000 UTC m=+3.104316771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs") pod "network-metrics-daemon-6rpfl" (UID: "af6ebd1b-3025-4a6f-9822-aa6606b8fa28") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:09.982057 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.982001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-run-openvswitch\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.982193 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.982086    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.982193 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.982141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-host-run-ovn-kubernetes\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.984671 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.984648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ca873b5-3619-4087-a38d-221452120a26-agent-certs\") pod \"konnectivity-agent-2qtbp\" (UID: \"0ca873b5-3619-4087-a38d-221452120a26\") " pod="kube-system/konnectivity-agent-2qtbp"
Sep 11 03:50:09.984773 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.984706    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-ovn-node-metrics-cert\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:09.986411 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:09.986297    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:50:09.986411 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:09.986322    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:50:09.986411 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:09.986338    2581 projected.go:196] Error preparing data for projected volume kube-api-access-j77rq for pod openshift-network-diagnostics/network-check-target-t7sgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:09.986411 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:09.986413    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq podName:7c125f86-ef79-4811-ab32-abacb2b62402 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:10.486396239 +0000 UTC m=+3.108789249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j77rq" (UniqueName: "kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq") pod "network-check-target-t7sgk" (UID: "7c125f86-ef79-4811-ab32-abacb2b62402") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:09.988005 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.987983    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jphws\" (UniqueName: \"kubernetes.io/projected/5604fe2c-0315-490c-a234-cae850ae2d55-kube-api-access-jphws\") pod \"node-resolver-4cm6g\" (UID: \"5604fe2c-0315-490c-a234-cae850ae2d55\") " pod="openshift-dns/node-resolver-4cm6g"
Sep 11 03:50:09.988378 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.988358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2dl\" (UniqueName: \"kubernetes.io/projected/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-kube-api-access-kp2dl\") pod \"network-metrics-daemon-6rpfl\" (UID: \"af6ebd1b-3025-4a6f-9822-aa6606b8fa28\") " pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:09.988702 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.988682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x652m\" (UniqueName: \"kubernetes.io/projected/657845f4-5ec8-4c63-9d6a-f9ca4273be0e-kube-api-access-x652m\") pod \"node-ca-mkxx2\" (UID: \"657845f4-5ec8-4c63-9d6a-f9ca4273be0e\") " pod="openshift-image-registry/node-ca-mkxx2"
Sep 11 03:50:09.989593 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:09.989570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hwcb\" (UniqueName: \"kubernetes.io/projected/8f89a78c-3f9c-45bf-a1dc-3544ed06740f-kube-api-access-4hwcb\") pod \"ovnkube-node-bxqls\" (UID: \"8f89a78c-3f9c-45bf-a1dc-3544ed06740f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:10.070542 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.070468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dpprn"
Sep 11 03:50:10.079158 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.079133    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2"
Sep 11 03:50:10.087860 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.087829    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kmrwl"
Sep 11 03:50:10.095553 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.095527    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vqrfz"
Sep 11 03:50:10.102152 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.102134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tvdds"
Sep 11 03:50:10.109646 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.109629    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4cm6g"
Sep 11 03:50:10.117403 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.117385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:10.125852 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.125837    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mkxx2"
Sep 11 03:50:10.129348 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.129329    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2qtbp"
Sep 11 03:50:10.483768 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.483693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs\") pod \"network-metrics-daemon-6rpfl\" (UID: \"af6ebd1b-3025-4a6f-9822-aa6606b8fa28\") " pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:10.483884 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.483779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:10.483884 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:10.483837    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:10.483950 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:10.483892    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:10.483950 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:10.483900    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs podName:af6ebd1b-3025-4a6f-9822-aa6606b8fa28 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:11.483884862 +0000 UTC m=+4.106277873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs") pod "network-metrics-daemon-6rpfl" (UID: "af6ebd1b-3025-4a6f-9822-aa6606b8fa28") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:10.484038 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:10.483959    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret podName:69e1dbc6-8f47-4733-9be1-d76b49f7f4d1 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:11.483942412 +0000 UTC m=+4.106335420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret") pod "global-pull-secret-syncer-6bs2d" (UID: "69e1dbc6-8f47-4733-9be1-d76b49f7f4d1") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:10.584431 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.584397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77rq\" (UniqueName: \"kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq\") pod \"network-check-target-t7sgk\" (UID: \"7c125f86-ef79-4811-ab32-abacb2b62402\") " pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:10.584608 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:10.584538    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:50:10.584608 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:10.584562    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:50:10.584608 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:10.584574    2581 projected.go:196] Error preparing data for projected volume kube-api-access-j77rq for pod openshift-network-diagnostics/network-check-target-t7sgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:10.584735 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:10.584629    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq podName:7c125f86-ef79-4811-ab32-abacb2b62402 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:11.584611412 +0000 UTC m=+4.207004416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j77rq" (UniqueName: "kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq") pod "network-check-target-t7sgk" (UID: "7c125f86-ef79-4811-ab32-abacb2b62402") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:10.802116 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.802072    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 03:45:09 +0000 UTC" deadline="2028-03-09 06:21:23.128842629 +0000 UTC"
Sep 11 03:50:10.802116 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.802114    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13082h31m12.326731715s"
Sep 11 03:50:10.814942 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:10.814910    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2d0186_ceb5_4b85_b5d1_3a6687fb8799.slice/crio-6df4241b32714061530b0c008f979db1a13858ca56e64d5b06020dc469a29037 WatchSource:0}: Error finding container 6df4241b32714061530b0c008f979db1a13858ca56e64d5b06020dc469a29037: Status 404 returned error can't find the container with id 6df4241b32714061530b0c008f979db1a13858ca56e64d5b06020dc469a29037
Sep 11 03:50:10.834815 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:10.834697    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5604fe2c_0315_490c_a234_cae850ae2d55.slice/crio-7585af427761f39d82595b4321d9fcc3f908ac7d7d937944ca54493a48cfd7ef WatchSource:0}: Error finding container 7585af427761f39d82595b4321d9fcc3f908ac7d7d937944ca54493a48cfd7ef: Status 404 returned error can't find the container with id 7585af427761f39d82595b4321d9fcc3f908ac7d7d937944ca54493a48cfd7ef
Sep 11 03:50:10.837764 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:10.837743    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f89a78c_3f9c_45bf_a1dc_3544ed06740f.slice/crio-effc17ccee81b26ce88bb2062c81ed20d3541d1d13d10456afc8977f17ae8a3d WatchSource:0}: Error finding container effc17ccee81b26ce88bb2062c81ed20d3541d1d13d10456afc8977f17ae8a3d: Status 404 returned error can't find the container with id effc17ccee81b26ce88bb2062c81ed20d3541d1d13d10456afc8977f17ae8a3d
Sep 11 03:50:10.842211 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:10.842191    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ad581b_7f8b_4747_8aad_7b59824edac5.slice/crio-538aa6bf1443f098ab3db0d765877e30403883e3555ac2d9ff354c85c6c2af3f WatchSource:0}: Error finding container 538aa6bf1443f098ab3db0d765877e30403883e3555ac2d9ff354c85c6c2af3f: Status 404 returned error can't find the container with id 538aa6bf1443f098ab3db0d765877e30403883e3555ac2d9ff354c85c6c2af3f
Sep 11 03:50:10.843314 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:10.843290    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802d5da6_9490_4950_8c1b_da0454e85ad8.slice/crio-3cd1866dfe241ec57cebbd1862f606e24e6af72b7e87c55a84e71e46a81e7a53 WatchSource:0}: Error finding container 3cd1866dfe241ec57cebbd1862f606e24e6af72b7e87c55a84e71e46a81e7a53: Status 404 returned error can't find the container with id 3cd1866dfe241ec57cebbd1862f606e24e6af72b7e87c55a84e71e46a81e7a53
Sep 11 03:50:10.844553 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:10.844525    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657845f4_5ec8_4c63_9d6a_f9ca4273be0e.slice/crio-6114f6edb93dbc96029b1c7ac74e375ae960a25371f5ee3a3aba1f13f48e4a36 WatchSource:0}: Error finding container 6114f6edb93dbc96029b1c7ac74e375ae960a25371f5ee3a3aba1f13f48e4a36: Status 404 returned error can't find the container with id 6114f6edb93dbc96029b1c7ac74e375ae960a25371f5ee3a3aba1f13f48e4a36
Sep 11 03:50:10.845703 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:50:10.845569    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673f4848_26b9_4885_8857_b0cb058adb40.slice/crio-40a03039e5166bfcd03b2bd27c5d42fed60050303af64af4132c27d4ac43c3d6 WatchSource:0}: Error finding container 40a03039e5166bfcd03b2bd27c5d42fed60050303af64af4132c27d4ac43c3d6: Status 404 returned error can't find the container with id 40a03039e5166bfcd03b2bd27c5d42fed60050303af64af4132c27d4ac43c3d6
Sep 11 03:50:10.855273 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.855165    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:10.855334 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:10.855275    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:10.862618 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.862593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vqrfz" event={"ID":"443eee6f-26ee-439c-bd85-2bc0b92fdd14","Type":"ContainerStarted","Data":"f1abacba523d49a1be87b02d7d152019e3086a1f26e04d51a23234fdd246750f"}
Sep 11 03:50:10.863819 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.863796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dpprn" event={"ID":"673f4848-26b9-4885-8857-b0cb058adb40","Type":"ContainerStarted","Data":"40a03039e5166bfcd03b2bd27c5d42fed60050303af64af4132c27d4ac43c3d6"}
Sep 11 03:50:10.865106 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.865010    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mkxx2" event={"ID":"657845f4-5ec8-4c63-9d6a-f9ca4273be0e","Type":"ContainerStarted","Data":"6114f6edb93dbc96029b1c7ac74e375ae960a25371f5ee3a3aba1f13f48e4a36"}
Sep 11 03:50:10.866070 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.866053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tvdds" event={"ID":"f5ad581b-7f8b-4747-8aad-7b59824edac5","Type":"ContainerStarted","Data":"538aa6bf1443f098ab3db0d765877e30403883e3555ac2d9ff354c85c6c2af3f"}
Sep 11 03:50:10.867188 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.867170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4cm6g" event={"ID":"5604fe2c-0315-490c-a234-cae850ae2d55","Type":"ContainerStarted","Data":"7585af427761f39d82595b4321d9fcc3f908ac7d7d937944ca54493a48cfd7ef"}
Sep 11 03:50:10.868386 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.868362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kmrwl" event={"ID":"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799","Type":"ContainerStarted","Data":"6df4241b32714061530b0c008f979db1a13858ca56e64d5b06020dc469a29037"}
Sep 11 03:50:10.869503 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.869485    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2" event={"ID":"802d5da6-9490-4950-8c1b-da0454e85ad8","Type":"ContainerStarted","Data":"3cd1866dfe241ec57cebbd1862f606e24e6af72b7e87c55a84e71e46a81e7a53"}
Sep 11 03:50:10.870906 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.870882    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls" event={"ID":"8f89a78c-3f9c-45bf-a1dc-3544ed06740f","Type":"ContainerStarted","Data":"effc17ccee81b26ce88bb2062c81ed20d3541d1d13d10456afc8977f17ae8a3d"}
Sep 11 03:50:10.871897 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:10.871877    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2qtbp" event={"ID":"0ca873b5-3619-4087-a38d-221452120a26","Type":"ContainerStarted","Data":"afb828d00262285ad5be29262e98bb1b9193fe758260c6e15cbb942690b71993"}
Sep 11 03:50:11.492475 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:11.492181    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:11.492636 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:11.492483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs\") pod \"network-metrics-daemon-6rpfl\" (UID: \"af6ebd1b-3025-4a6f-9822-aa6606b8fa28\") " pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:11.492636 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:11.492602    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:11.492741 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:11.492659    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs podName:af6ebd1b-3025-4a6f-9822-aa6606b8fa28 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:13.492640179 +0000 UTC m=+6.115033194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs") pod "network-metrics-daemon-6rpfl" (UID: "af6ebd1b-3025-4a6f-9822-aa6606b8fa28") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:11.493058 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:11.493041    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:11.493145 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:11.493092    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret podName:69e1dbc6-8f47-4733-9be1-d76b49f7f4d1 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:13.493076703 +0000 UTC m=+6.115469715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret") pod "global-pull-secret-syncer-6bs2d" (UID: "69e1dbc6-8f47-4733-9be1-d76b49f7f4d1") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:11.592845 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:11.592807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77rq\" (UniqueName: \"kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq\") pod \"network-check-target-t7sgk\" (UID: \"7c125f86-ef79-4811-ab32-abacb2b62402\") " pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:11.593052 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:11.593035    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:50:11.593122 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:11.593060    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:50:11.593122 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:11.593072    2581 projected.go:196] Error preparing data for projected volume kube-api-access-j77rq for pod openshift-network-diagnostics/network-check-target-t7sgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:11.593222 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:11.593127    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq podName:7c125f86-ef79-4811-ab32-abacb2b62402 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:13.593110584 +0000 UTC m=+6.215503596 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j77rq" (UniqueName: "kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq") pod "network-check-target-t7sgk" (UID: "7c125f86-ef79-4811-ab32-abacb2b62402") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:11.803149 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:11.803065    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 03:45:09 +0000 UTC" deadline="2028-04-21 04:00:16.879188606 +0000 UTC"
Sep 11 03:50:11.803149 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:11.803103    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14112h10m5.076090748s"
Sep 11 03:50:11.857800 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:11.857769    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:11.857968 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:11.857943    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:11.858382 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:11.858361    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:11.858567 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:11.858485    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:11.883043 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:11.882117    2581 generic.go:356] "Generic (PLEG): container finished" podID="79d681e26c169b5ac07eacf2cb3d45e4" containerID="1d471a0b6f081850193a45d21de55c880eb03457475a6148db68e37be611b9da" exitCode=0
Sep 11 03:50:11.883043 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:11.882843    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal" event={"ID":"79d681e26c169b5ac07eacf2cb3d45e4","Type":"ContainerDied","Data":"1d471a0b6f081850193a45d21de55c880eb03457475a6148db68e37be611b9da"}
Sep 11 03:50:11.893391 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:11.893362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal" event={"ID":"427abdc7e2784c3c5f6bbe0ee342a5b6","Type":"ContainerStarted","Data":"dbdc022764b71bb870fc7ab48a1cfe3c0311ac9e3a79230513c94474733943b9"}
Sep 11 03:50:12.855587 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:12.855554    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:12.856189 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:12.855694    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:12.912977 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:12.912939    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal" event={"ID":"79d681e26c169b5ac07eacf2cb3d45e4","Type":"ContainerStarted","Data":"d88694169df53d60c6ecd49fbe2cfc24f91f2579daffa18b605efcffa1105468"}
Sep 11 03:50:12.929779 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:12.929662    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal" podStartSLOduration=4.929639551 podStartE2EDuration="4.929639551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:11.919311595 +0000 UTC m=+4.541704624" watchObservedRunningTime="2026-09-11 03:50:12.929639551 +0000 UTC m=+5.552032579"
Sep 11 03:50:13.510242 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:13.509436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:13.510242 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:13.509525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs\") pod \"network-metrics-daemon-6rpfl\" (UID: \"af6ebd1b-3025-4a6f-9822-aa6606b8fa28\") " pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:13.510242 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:13.509661    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:13.510242 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:13.509724    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs podName:af6ebd1b-3025-4a6f-9822-aa6606b8fa28 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:17.509705951 +0000 UTC m=+10.132098962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs") pod "network-metrics-daemon-6rpfl" (UID: "af6ebd1b-3025-4a6f-9822-aa6606b8fa28") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:13.510242 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:13.510146    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:13.510242 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:13.510206    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret podName:69e1dbc6-8f47-4733-9be1-d76b49f7f4d1 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:17.51019104 +0000 UTC m=+10.132584052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret") pod "global-pull-secret-syncer-6bs2d" (UID: "69e1dbc6-8f47-4733-9be1-d76b49f7f4d1") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:13.610999 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:13.610337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77rq\" (UniqueName: \"kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq\") pod \"network-check-target-t7sgk\" (UID: \"7c125f86-ef79-4811-ab32-abacb2b62402\") " pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:13.610999 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:13.610509    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:50:13.610999 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:13.610531    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:50:13.610999 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:13.610544    2581 projected.go:196] Error preparing data for projected volume kube-api-access-j77rq for pod openshift-network-diagnostics/network-check-target-t7sgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:13.610999 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:13.610598    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq podName:7c125f86-ef79-4811-ab32-abacb2b62402 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:17.610581225 +0000 UTC m=+10.232974237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j77rq" (UniqueName: "kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq") pod "network-check-target-t7sgk" (UID: "7c125f86-ef79-4811-ab32-abacb2b62402") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:13.856736 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:13.856179    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:13.856736 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:13.856205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:13.856736 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:13.856320    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:13.856736 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:13.856460    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:14.855299 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:14.855261    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:14.855479 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:14.855389    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:15.855771 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:15.855734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:15.856405 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:15.855880    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:15.856405 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:15.855951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:15.856405 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:15.856096    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:16.855373 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:16.855295    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:16.855544 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:16.855437    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:17.542193 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:17.542153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs\") pod \"network-metrics-daemon-6rpfl\" (UID: \"af6ebd1b-3025-4a6f-9822-aa6606b8fa28\") " pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:17.542589 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:17.542228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:17.542589 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:17.542352    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:17.542589 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:17.542365    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:17.542589 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:17.542420    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs podName:af6ebd1b-3025-4a6f-9822-aa6606b8fa28 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:25.542404934 +0000 UTC m=+18.164797939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs") pod "network-metrics-daemon-6rpfl" (UID: "af6ebd1b-3025-4a6f-9822-aa6606b8fa28") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:17.542589 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:17.542438    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret podName:69e1dbc6-8f47-4733-9be1-d76b49f7f4d1 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:25.542428307 +0000 UTC m=+18.164821336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret") pod "global-pull-secret-syncer-6bs2d" (UID: "69e1dbc6-8f47-4733-9be1-d76b49f7f4d1") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:17.642809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:17.642774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77rq\" (UniqueName: \"kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq\") pod \"network-check-target-t7sgk\" (UID: \"7c125f86-ef79-4811-ab32-abacb2b62402\") " pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:17.642983 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:17.642967    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:50:17.643071 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:17.643031    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:50:17.643125 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:17.643073    2581 projected.go:196] Error preparing data for projected volume kube-api-access-j77rq for pod openshift-network-diagnostics/network-check-target-t7sgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:17.643210 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:17.643195    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq podName:7c125f86-ef79-4811-ab32-abacb2b62402 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:25.643156797 +0000 UTC m=+18.265549814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j77rq" (UniqueName: "kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq") pod "network-check-target-t7sgk" (UID: "7c125f86-ef79-4811-ab32-abacb2b62402") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:17.856349 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:17.856227    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:17.856349 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:17.856335    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:17.856648 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:17.856574    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:17.856733 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:17.856694    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:18.855385 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:18.855348    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:18.855872 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:18.855491    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:19.855189 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:19.855102    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:19.855189 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:19.855134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:19.855436 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:19.855243    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:19.855914 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:19.855688    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:20.855260 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:20.855233    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:20.855421 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:20.855322    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:21.855863 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:21.855830    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:21.856352 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:21.855968    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:21.856352 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:21.856049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:21.856352 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:21.856165    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:22.855800 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:22.855768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:22.855961 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:22.855877    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:23.855202 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:23.855165    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:23.855391 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:23.855311    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:23.855391 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:23.855350    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:23.855540 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:23.855439    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:24.855773 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:24.855745    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:24.856281 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:24.855834    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:25.597299 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:25.597257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs\") pod \"network-metrics-daemon-6rpfl\" (UID: \"af6ebd1b-3025-4a6f-9822-aa6606b8fa28\") " pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:25.597477 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:25.597328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:25.597477 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:25.597395    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:25.597477 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:25.597464    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:25.597608 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:25.597480    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs podName:af6ebd1b-3025-4a6f-9822-aa6606b8fa28 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:41.597457081 +0000 UTC m=+34.219850119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs") pod "network-metrics-daemon-6rpfl" (UID: "af6ebd1b-3025-4a6f-9822-aa6606b8fa28") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:25.597608 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:25.597507    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret podName:69e1dbc6-8f47-4733-9be1-d76b49f7f4d1 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:41.597495245 +0000 UTC m=+34.219888255 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret") pod "global-pull-secret-syncer-6bs2d" (UID: "69e1dbc6-8f47-4733-9be1-d76b49f7f4d1") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:25.697978 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:25.697941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77rq\" (UniqueName: \"kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq\") pod \"network-check-target-t7sgk\" (UID: \"7c125f86-ef79-4811-ab32-abacb2b62402\") " pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:25.698168 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:25.698099    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:50:25.698168 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:25.698117    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:50:25.698168 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:25.698126    2581 projected.go:196] Error preparing data for projected volume kube-api-access-j77rq for pod openshift-network-diagnostics/network-check-target-t7sgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:25.698291 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:25.698176    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq podName:7c125f86-ef79-4811-ab32-abacb2b62402 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:41.698164006 +0000 UTC m=+34.320557012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j77rq" (UniqueName: "kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq") pod "network-check-target-t7sgk" (UID: "7c125f86-ef79-4811-ab32-abacb2b62402") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:25.855658 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:25.855581    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:25.855816 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:25.855704    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:25.855816 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:25.855748    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:25.856296 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:25.855874    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:26.855507 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:26.855473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:26.855675 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:26.855581    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:27.855858 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:27.855825    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:27.856320 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:27.855956    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:27.856320 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:27.856030    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:27.856320 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:27.856142    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:28.855576 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.855275    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:28.855703 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:28.855681    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:28.945922 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.945891    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2qtbp" event={"ID":"0ca873b5-3619-4087-a38d-221452120a26","Type":"ContainerStarted","Data":"9ba04e6f5ef6baea9055cf9819321fcc00d6064f8b6521ad8e6a7833246429bf"}
Sep 11 03:50:28.947709 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.947668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vqrfz" event={"ID":"443eee6f-26ee-439c-bd85-2bc0b92fdd14","Type":"ContainerStarted","Data":"3a8ecbffcd3f16286304a160c2dd128d674e0dcae60d0695389c238b876f1d56"}
Sep 11 03:50:28.949776 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.949749    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mkxx2" event={"ID":"657845f4-5ec8-4c63-9d6a-f9ca4273be0e","Type":"ContainerStarted","Data":"a42141d5adf348d6e97577d8b90c19a0c03955244c83c1882261553a47ac754a"}
Sep 11 03:50:28.951571 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.951541    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5ad581b-7f8b-4747-8aad-7b59824edac5" containerID="f89211d1f98b6d610d77adfe8d0c2432bc47b396ba942e98cb0ce39ef6784e66" exitCode=0
Sep 11 03:50:28.951670 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.951612    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tvdds" event={"ID":"f5ad581b-7f8b-4747-8aad-7b59824edac5","Type":"ContainerDied","Data":"f89211d1f98b6d610d77adfe8d0c2432bc47b396ba942e98cb0ce39ef6784e66"}
Sep 11 03:50:28.953130 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.953110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4cm6g" event={"ID":"5604fe2c-0315-490c-a234-cae850ae2d55","Type":"ContainerStarted","Data":"ac32af07a07452bb24951801c50795f84e6d49c581e0ec55a1fe95fc8fce91d2"}
Sep 11 03:50:28.954685 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.954663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kmrwl" event={"ID":"9d2d0186-ceb5-4b85-b5d1-3a6687fb8799","Type":"ContainerStarted","Data":"6843f55948969ad80d690d18a2269f34da58045597b4335f540c769045c536ea"}
Sep 11 03:50:28.956143 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.956121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2" event={"ID":"802d5da6-9490-4950-8c1b-da0454e85ad8","Type":"ContainerStarted","Data":"38c82b898fa1e4dae36c84d7db587e3fb05ea3bf1f613ec9a5dcf24c55b31eab"}
Sep 11 03:50:28.960886 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.960868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls" event={"ID":"8f89a78c-3f9c-45bf-a1dc-3544ed06740f","Type":"ContainerStarted","Data":"9dedfd0720e0c506690e2eb03d2bdf3fe15b6215e054815beb70460c56795804"}
Sep 11 03:50:28.960973 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.960925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls" event={"ID":"8f89a78c-3f9c-45bf-a1dc-3544ed06740f","Type":"ContainerStarted","Data":"6dbb1fb3bc440966e42fed74a5049986eeebb89f517fa1ea56713b0e841fd30f"}
Sep 11 03:50:28.960973 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.960955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls" event={"ID":"8f89a78c-3f9c-45bf-a1dc-3544ed06740f","Type":"ContainerStarted","Data":"b89284940830c69089be58849d8587297ac17d15f44da5a87595887d0d13d8f1"}
Sep 11 03:50:28.961063 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.960990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls" event={"ID":"8f89a78c-3f9c-45bf-a1dc-3544ed06740f","Type":"ContainerStarted","Data":"bd98e51538cd9308adebf89e2ded07e7be650fe67adce9898274b715c97ee08e"}
Sep 11 03:50:28.961063 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.961035    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls" event={"ID":"8f89a78c-3f9c-45bf-a1dc-3544ed06740f","Type":"ContainerStarted","Data":"ff2232a694fd9cc76587b6c617c12a2c1a749430d25ebd507cff0742ebd4431a"}
Sep 11 03:50:28.961132 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.961067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls" event={"ID":"8f89a78c-3f9c-45bf-a1dc-3544ed06740f","Type":"ContainerStarted","Data":"97ed6870bc4439aa3b106b1d7b1a93a8c0e00fbad7ba6e4c0c1e27cc5072d064"}
Sep 11 03:50:28.962484 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.962443    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2qtbp" podStartSLOduration=3.494123007 podStartE2EDuration="20.962432231s" totalImagesPullingTime="17.468309224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:28.962206941 +0000 UTC m=+21.584599968" watchObservedRunningTime="2026-09-11 03:50:28.962432231 +0000 UTC m=+21.584825257"
Sep 11 03:50:28.962640 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.962622    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal" podStartSLOduration=20.962617111 podStartE2EDuration="20.962617111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:12.930079986 +0000 UTC m=+5.552473033" watchObservedRunningTime="2026-09-11 03:50:28.962617111 +0000 UTC m=+21.585010137"
Sep 11 03:50:28.977904 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:28.977871    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4cm6g" podStartSLOduration=4.489720666 podStartE2EDuration="21.977859565s" totalImagesPullingTime="17.488138899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:28.977749549 +0000 UTC m=+21.600142594" watchObservedRunningTime="2026-09-11 03:50:28.977859565 +0000 UTC m=+21.600252592"
Sep 11 03:50:29.001312 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.001277    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mkxx2" podStartSLOduration=3.523792176 podStartE2EDuration="21.001266048s" totalImagesPullingTime="17.477473872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:29.000821506 +0000 UTC m=+21.623214567" watchObservedRunningTime="2026-09-11 03:50:29.001266048 +0000 UTC m=+21.623659109"
Sep 11 03:50:29.029174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.029132    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kmrwl" podStartSLOduration=4.543583751 podStartE2EDuration="22.029117509s" totalImagesPullingTime="17.485533758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:29.028703125 +0000 UTC m=+21.651096152" watchObservedRunningTime="2026-09-11 03:50:29.029117509 +0000 UTC m=+21.651510537"
Sep 11 03:50:29.048935 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.048861    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vqrfz" podStartSLOduration=4.524238003 podStartE2EDuration="22.048847215s" totalImagesPullingTime="17.524609212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:29.048486247 +0000 UTC m=+21.670879264" watchObservedRunningTime="2026-09-11 03:50:29.048847215 +0000 UTC m=+21.671240241"
Sep 11 03:50:29.111465 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.111428    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:29.215634 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.215597    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:29.294758 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.294730    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:29.394174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.394138    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:29.826654 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.826633    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 03:50:29.827775 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.827694    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T03:50:29.826651062Z","UUID":"e269e876-e7bc-4db3-aa09-040600628d5c","Handler":null,"Name":"","Endpoint":""}
Sep 11 03:50:29.829150 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.829121    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 03:50:29.829239 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.829156    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 03:50:29.855970 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.855945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:29.856118 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.855956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:29.856177 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:29.856115    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:29.856235 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:29.856208    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:29.914268 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.914231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:29.964521 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.964445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dpprn" event={"ID":"673f4848-26b9-4885-8857-b0cb058adb40","Type":"ContainerStarted","Data":"6169cae4d295ab6dd8890a98b0f28a622d85a881f9fd1af7c694fa3ea59cde16"}
Sep 11 03:50:29.966558 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.966534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2" event={"ID":"802d5da6-9490-4950-8c1b-da0454e85ad8","Type":"ContainerStarted","Data":"653fcab70d37e950e8dd55a54ed528ea120c8fcaf4c85612e89b19ad61647527"}
Sep 11 03:50:29.968009 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.967981    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:29.978473 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:29.978426    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dpprn" podStartSLOduration=5.494447481 podStartE2EDuration="22.97841375s" totalImagesPullingTime="17.483966269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:29.978160281 +0000 UTC m=+22.600553330" watchObservedRunningTime="2026-09-11 03:50:29.97841375 +0000 UTC m=+22.600806799"
Sep 11 03:50:30.113296 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:30.113264    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:30.213467 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:30.213433    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:30.508264 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:30.508231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:30.572168 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:30.572120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:30.855112 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:30.855039    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:30.855263 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:30.855164    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:31.855971 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:31.855938    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:31.856506 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:31.855936    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:31.856506 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:31.856094    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:31.856506 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:31.856154    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:31.974223 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:31.974165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2" event={"ID":"802d5da6-9490-4950-8c1b-da0454e85ad8","Type":"ContainerStarted","Data":"76c9122b7f385b815549d8fd16fd454c6df3b527c2ccca16874b43a3d72e8324"}
Sep 11 03:50:31.978073 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:31.978047    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls" event={"ID":"8f89a78c-3f9c-45bf-a1dc-3544ed06740f","Type":"ContainerStarted","Data":"d255bdbd5a2f46c659bb17aaa52adad41d1c1646ffdd9ac2caba8b97942cd897"}
Sep 11 03:50:32.000275 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:32.000225    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb9r2" podStartSLOduration=5.295829815 podStartE2EDuration="25.00021075s" totalImagesPullingTime="19.704380935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:31.999543958 +0000 UTC m=+24.621937001" watchObservedRunningTime="2026-09-11 03:50:32.00021075 +0000 UTC m=+24.622603777"
Sep 11 03:50:32.038750 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:32.038548    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:32.139568 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:32.139441    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:32.855490 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:32.855459    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:32.855670 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:32.855602    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:32.874733 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:32.874693    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:32.944393 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:32.944356    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:33.394507 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:33.394421    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2qtbp"
Sep 11 03:50:33.395424 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:33.395394    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2qtbp"
Sep 11 03:50:33.855562 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:33.855534    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:33.855663 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:33.855535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:33.855663 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:33.855642    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:33.855762 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:33.855741    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:33.986087 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:33.985808    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls" event={"ID":"8f89a78c-3f9c-45bf-a1dc-3544ed06740f","Type":"ContainerStarted","Data":"cfdafe7721c4d819d27f3511f410bce86c7abd88549e45b43b0777942bb4f0ab"}
Sep 11 03:50:33.986555 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:33.986098    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:33.991668 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:33.991641    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5ad581b-7f8b-4747-8aad-7b59824edac5" containerID="443bb4866e1c5892015a794d6a194b95f7190999346685eb5b9825f34420af01" exitCode=0
Sep 11 03:50:33.991791 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:33.991733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tvdds" event={"ID":"f5ad581b-7f8b-4747-8aad-7b59824edac5","Type":"ContainerDied","Data":"443bb4866e1c5892015a794d6a194b95f7190999346685eb5b9825f34420af01"}
Sep 11 03:50:33.991939 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:33.991912    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2qtbp"
Sep 11 03:50:33.992408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:33.992371    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2qtbp"
Sep 11 03:50:34.001917 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:34.001893    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:34.013240 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:34.013200    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls" podStartSLOduration=8.461111952 podStartE2EDuration="26.013187751s" totalImagesPullingTime="17.552075799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:34.011554801 +0000 UTC m=+26.633947828" watchObservedRunningTime="2026-09-11 03:50:34.013187751 +0000 UTC m=+26.635580777"
Sep 11 03:50:34.101118 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:34.101077    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:34.855494 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:34.855468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:34.855711 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:34.855564    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:34.993623 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:34.993597    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 03:50:34.994102 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:34.993985    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:35.008348 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:35.008325    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:35.391207 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:35.391109    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6bs2d"]
Sep 11 03:50:35.391352 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:35.391229    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:35.391352 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:35.391336    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:35.391876 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:35.391852    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6rpfl"]
Sep 11 03:50:35.391983 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:35.391970    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:35.392590 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:35.392556    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:35.395988 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:35.394473    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t7sgk"]
Sep 11 03:50:35.395988 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:35.394614    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:35.395988 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:35.394718    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:35.997894 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:35.997863    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5ad581b-7f8b-4747-8aad-7b59824edac5" containerID="547f3cc93f3a8a1c10377ec2fcd73fa6af981df91809a222881c12bf0c014991" exitCode=0
Sep 11 03:50:35.998703 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:35.997952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tvdds" event={"ID":"f5ad581b-7f8b-4747-8aad-7b59824edac5","Type":"ContainerDied","Data":"547f3cc93f3a8a1c10377ec2fcd73fa6af981df91809a222881c12bf0c014991"}
Sep 11 03:50:35.998703 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:35.998115    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 03:50:36.657880 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:36.657851    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:36.758378 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:36.758302    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:36.855661 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:36.855633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:36.855661 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:36.855667    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:36.855862 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:36.855747    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:36.855922 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:36.855897    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:37.000667 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:37.000637    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 03:50:37.233563 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:37.233015    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:37.292724 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:37.292680    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:37.393006 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:37.392975    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:37.452986 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:37.452918    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:37.855801 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:37.855584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:37.855801 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:37.855787    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:38.004830 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:38.004799    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5ad581b-7f8b-4747-8aad-7b59824edac5" containerID="47fa24553f18530a03489caf7bfa677e4a1f9b1b76861dec57af65f6024aba18" exitCode=0
Sep 11 03:50:38.005303 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:38.004856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tvdds" event={"ID":"f5ad581b-7f8b-4747-8aad-7b59824edac5","Type":"ContainerDied","Data":"47fa24553f18530a03489caf7bfa677e4a1f9b1b76861dec57af65f6024aba18"}
Sep 11 03:50:38.855887 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:38.855852    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:38.856065 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:38.855852    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:38.856065 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:38.855991    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:38.856185 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:38.856066    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:39.002160 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:39.002128    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:39.373178 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:39.373146    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:50:39.373627 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:39.373392    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 03:50:39.384589 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:39.384549    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls" podUID="8f89a78c-3f9c-45bf-a1dc-3544ed06740f" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 11 03:50:39.392826 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:39.392798    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls" podUID="8f89a78c-3f9c-45bf-a1dc-3544ed06740f" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 11 03:50:39.855125 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:39.855097    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:39.855305 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:39.855204    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:39.891680 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:39.891646    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:40.855538 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:40.855508    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:40.856072 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:40.855508    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:40.856072 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:40.855642    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:40.856072 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:40.855735    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:41.618502 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:41.618414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:41.618502 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:41.618498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs\") pod \"network-metrics-daemon-6rpfl\" (UID: \"af6ebd1b-3025-4a6f-9822-aa6606b8fa28\") " pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:41.618697 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:41.618542    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:41.618697 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:41.618589    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:41.618697 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:41.618610    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret podName:69e1dbc6-8f47-4733-9be1-d76b49f7f4d1 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:13.618594202 +0000 UTC m=+66.240987206 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret") pod "global-pull-secret-syncer-6bs2d" (UID: "69e1dbc6-8f47-4733-9be1-d76b49f7f4d1") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:41.618697 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:41.618629    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs podName:af6ebd1b-3025-4a6f-9822-aa6606b8fa28 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:13.618618339 +0000 UTC m=+66.241011344 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs") pod "network-metrics-daemon-6rpfl" (UID: "af6ebd1b-3025-4a6f-9822-aa6606b8fa28") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:41.719760 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:41.719718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77rq\" (UniqueName: \"kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq\") pod \"network-check-target-t7sgk\" (UID: \"7c125f86-ef79-4811-ab32-abacb2b62402\") " pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:41.719922 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:41.719883    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:50:41.719922 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:41.719905    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:50:41.719922 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:41.719919    2581 projected.go:196] Error preparing data for projected volume kube-api-access-j77rq for pod openshift-network-diagnostics/network-check-target-t7sgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:41.720068 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:41.719979    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq podName:7c125f86-ef79-4811-ab32-abacb2b62402 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:13.719964691 +0000 UTC m=+66.342357696 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j77rq" (UniqueName: "kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq") pod "network-check-target-t7sgk" (UID: "7c125f86-ef79-4811-ab32-abacb2b62402") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:41.759255 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:41.759205    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:41.855080 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:41.855044    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:41.855248 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:41.855177    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:41.883151 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:41.883057    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:42.855857 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:42.855826    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:42.855857 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:42.855858    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:42.856063 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:42.855947    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:42.856106 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:42.856082    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:42.985795 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:42.985759    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:43.019876 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:43.019846    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5ad581b-7f8b-4747-8aad-7b59824edac5" containerID="5b527c8b0c4c1e216723606c4bb357b9e4eef4e0cecac4ea3f4101546bfa1560" exitCode=0
Sep 11 03:50:43.020010 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:43.019897    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tvdds" event={"ID":"f5ad581b-7f8b-4747-8aad-7b59824edac5","Type":"ContainerDied","Data":"5b527c8b0c4c1e216723606c4bb357b9e4eef4e0cecac4ea3f4101546bfa1560"}
Sep 11 03:50:43.394790 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:43.394703    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:43.508748 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:43.508716    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:43.855431 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:43.855402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:43.855605 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:43.855502    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:43.898206 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:43.898175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:44.001985 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:44.001957    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:44.028875 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:44.028849    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5ad581b-7f8b-4747-8aad-7b59824edac5" containerID="fd77f53b926650e28772841f9e927949d4a2364756b90ab865c2c8587b850571" exitCode=0
Sep 11 03:50:44.028993 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:44.028886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tvdds" event={"ID":"f5ad581b-7f8b-4747-8aad-7b59824edac5","Type":"ContainerDied","Data":"fd77f53b926650e28772841f9e927949d4a2364756b90ab865c2c8587b850571"}
Sep 11 03:50:44.098264 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:44.098231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:44.197538 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:44.197505    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:44.497238 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:44.497164    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:44.855203 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:44.855174    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:44.855203 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:44.855200    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:44.855404 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:44.855269    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:44.855404 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:44.855375    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:45.033803 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:45.033774    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tvdds" event={"ID":"f5ad581b-7f8b-4747-8aad-7b59824edac5","Type":"ContainerStarted","Data":"1db4a0640a1d5964ae6bdbf4514db99b822a56d2e40426bb18b3ad2b9b9e1f69"}
Sep 11 03:50:45.061032 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:45.060987    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tvdds" podStartSLOduration=9.331121911 podStartE2EDuration="38.060967483s" totalImagesPullingTime="28.729845572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:45.059234565 +0000 UTC m=+37.681627591" watchObservedRunningTime="2026-09-11 03:50:45.060967483 +0000 UTC m=+37.683360511"
Sep 11 03:50:45.293463 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:45.293433    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:45.691149 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:45.691113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:45.855298 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:45.855268    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:45.855479 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:45.855370    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:45.869681 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:45.869651    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:45.920212 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:45.920181    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:46.094846 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:46.094820    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:46.297087 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:46.297048    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:46.400583 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:46.400494    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:46.488405 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:46.488376    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:46.855078 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:46.855051    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:46.855217 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:46.855150    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:46.855298 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:46.855214    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:46.855357 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:46.855326    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:47.100958 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:47.100910    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:47.296225 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:47.296133    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:47.694645 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:47.694604    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:47.856298 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:47.856270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:47.856445 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:47.856349    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:47.891294 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:47.891246    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:48.100743 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:48.100714    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:48.148344 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:48.148316    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:48.292564 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:48.292531    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:48.493904 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:48.493833    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:48.855156 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:48.855128    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:48.855328 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:48.855275    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:48.855328 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:48.855314    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:48.855455 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:48.855382    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:49.492140 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:49.492112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:49.739509 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:49.739474    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:49.794104 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:49.794013    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:49.855213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:49.855183    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:49.855350 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:49.855297    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:50.102806 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:50.102777    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:50.294125 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:50.294095    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:50.693368 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:50.693337    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:50.855434 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:50.855404    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:50.855434 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:50.855435    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:50.855633 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:50.855521    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:50.855668 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:50.855637    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:51.292314 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:51.292279    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:51.696221 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:51.696189    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:51.845359 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:51.845328    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:51.855282 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:51.855259    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:51.855413 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:51.855347    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:51.955603 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:51.955514    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:52.101417 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:52.101383    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:52.298687 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:52.298599    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:52.354735 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:52.354705    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:52.527555 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:52.527525    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:52.593209 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:52.593183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:52.700723 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:52.700687    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:52.804687 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:52.804654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:52.855224 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:52.855135    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:52.855395 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:52.855254    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:52.855395 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:52.855314    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:52.855519 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:52.855426    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:53.096548 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:53.096518    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:53.496649 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:53.496562    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:53.597956 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:53.597925    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:53.697476 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:53.697442    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:53.798431 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:53.798347    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:53.855658 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:53.855630    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:53.855803 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:53.855748    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:53.931844 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:53.931811    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:54.094266 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:54.094235    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:54.557128 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:54.557096    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:54.657777 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:54.657751    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:50:54.855444 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:54.855359    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:54.855788 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:54.855359    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:54.855788 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:54.855487    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:54.855788 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:54.855527    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:55.855855 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:55.855821    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:55.856262 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:55.855919    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:56.855079 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:56.855044    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:56.855079 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:56.855074    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:56.855363 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:56.855164    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:56.855363 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:56.855277    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:57.856531 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:57.856502    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:57.856896 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:57.856576    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:50:58.855152 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:58.855120    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:50:58.855341 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:58.855120    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:50:58.855341 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:58.855325    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:50:58.855445 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:58.855212    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:50:59.855853 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:50:59.855825    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:50:59.856265 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:50:59.855918    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:51:00.855071 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:00.855041    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:51:00.855238 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:00.855142    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6rpfl" podUID="af6ebd1b-3025-4a6f-9822-aa6606b8fa28"
Sep 11 03:51:00.855238 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:00.855197    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:51:00.855309 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:00.855280    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bs2d" podUID="69e1dbc6-8f47-4733-9be1-d76b49f7f4d1"
Sep 11 03:51:01.582989 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:01.582955    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:01.670034 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:01.669985    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:01.769545 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:01.769512    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:01.855696 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:01.855675    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:51:01.855799 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:01.855767    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sgk" podUID="7c125f86-ef79-4811-ab32-abacb2b62402"
Sep 11 03:51:01.868172 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:01.868145    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:01.969089 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:01.969050    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:02.039705 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.039673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:02.157754 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.157684    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeReady"
Sep 11 03:51:02.157878 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.157802    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 03:51:02.204377 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.204347    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5gtpq"]
Sep 11 03:51:02.206235 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.206219    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:02.207422 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.207403    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-65vbf"]
Sep 11 03:51:02.208868 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.208852    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hznd4\""
Sep 11 03:51:02.209041 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.209010    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-65vbf"
Sep 11 03:51:02.209101 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.209061    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 03:51:02.212015 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.209672    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 03:51:02.212582 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.212559    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 03:51:02.212776 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.212634    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g2lkd\""
Sep 11 03:51:02.212881 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.212663    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 03:51:02.213033 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.213005    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 03:51:02.218194 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.218174    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5gtpq"]
Sep 11 03:51:02.221185 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.221167    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-65vbf"]
Sep 11 03:51:02.274482 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.274459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e7b003d-b66d-4eef-8738-29ae74b53cc5-config-volume\") pod \"dns-default-5gtpq\" (UID: \"1e7b003d-b66d-4eef-8738-29ae74b53cc5\") " pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:02.274568 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.274490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb5fr\" (UniqueName: \"kubernetes.io/projected/40496107-347a-4a7d-bf59-25f0f8ac4a41-kube-api-access-xb5fr\") pod \"ingress-canary-65vbf\" (UID: \"40496107-347a-4a7d-bf59-25f0f8ac4a41\") " pod="openshift-ingress-canary/ingress-canary-65vbf"
Sep 11 03:51:02.274568 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.274507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7b003d-b66d-4eef-8738-29ae74b53cc5-tmp-dir\") pod \"dns-default-5gtpq\" (UID: \"1e7b003d-b66d-4eef-8738-29ae74b53cc5\") " pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:02.274568 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.274521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfh5\" (UniqueName: \"kubernetes.io/projected/1e7b003d-b66d-4eef-8738-29ae74b53cc5-kube-api-access-rvfh5\") pod \"dns-default-5gtpq\" (UID: \"1e7b003d-b66d-4eef-8738-29ae74b53cc5\") " pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:02.274568 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.274538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e7b003d-b66d-4eef-8738-29ae74b53cc5-metrics-tls\") pod \"dns-default-5gtpq\" (UID: \"1e7b003d-b66d-4eef-8738-29ae74b53cc5\") " pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:02.274695 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.274591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40496107-347a-4a7d-bf59-25f0f8ac4a41-cert\") pod \"ingress-canary-65vbf\" (UID: \"40496107-347a-4a7d-bf59-25f0f8ac4a41\") " pod="openshift-ingress-canary/ingress-canary-65vbf"
Sep 11 03:51:02.375807 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.375772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e7b003d-b66d-4eef-8738-29ae74b53cc5-config-volume\") pod \"dns-default-5gtpq\" (UID: \"1e7b003d-b66d-4eef-8738-29ae74b53cc5\") " pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:02.375921 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.375815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb5fr\" (UniqueName: \"kubernetes.io/projected/40496107-347a-4a7d-bf59-25f0f8ac4a41-kube-api-access-xb5fr\") pod \"ingress-canary-65vbf\" (UID: \"40496107-347a-4a7d-bf59-25f0f8ac4a41\") " pod="openshift-ingress-canary/ingress-canary-65vbf"
Sep 11 03:51:02.375921 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.375829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7b003d-b66d-4eef-8738-29ae74b53cc5-tmp-dir\") pod \"dns-default-5gtpq\" (UID: \"1e7b003d-b66d-4eef-8738-29ae74b53cc5\") " pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:02.375921 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.375845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfh5\" (UniqueName: \"kubernetes.io/projected/1e7b003d-b66d-4eef-8738-29ae74b53cc5-kube-api-access-rvfh5\") pod \"dns-default-5gtpq\" (UID: \"1e7b003d-b66d-4eef-8738-29ae74b53cc5\") " pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:02.375921 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.375866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e7b003d-b66d-4eef-8738-29ae74b53cc5-metrics-tls\") pod \"dns-default-5gtpq\" (UID: \"1e7b003d-b66d-4eef-8738-29ae74b53cc5\") " pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:02.376125 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.376011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40496107-347a-4a7d-bf59-25f0f8ac4a41-cert\") pod \"ingress-canary-65vbf\" (UID: \"40496107-347a-4a7d-bf59-25f0f8ac4a41\") " pod="openshift-ingress-canary/ingress-canary-65vbf"
Sep 11 03:51:02.376293 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.376275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7b003d-b66d-4eef-8738-29ae74b53cc5-tmp-dir\") pod \"dns-default-5gtpq\" (UID: \"1e7b003d-b66d-4eef-8738-29ae74b53cc5\") " pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:02.376424 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.376407    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e7b003d-b66d-4eef-8738-29ae74b53cc5-config-volume\") pod \"dns-default-5gtpq\" (UID: \"1e7b003d-b66d-4eef-8738-29ae74b53cc5\") " pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:02.379644 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.379616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e7b003d-b66d-4eef-8738-29ae74b53cc5-metrics-tls\") pod \"dns-default-5gtpq\" (UID: \"1e7b003d-b66d-4eef-8738-29ae74b53cc5\") " pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:02.379829 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.379807    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40496107-347a-4a7d-bf59-25f0f8ac4a41-cert\") pod \"ingress-canary-65vbf\" (UID: \"40496107-347a-4a7d-bf59-25f0f8ac4a41\") " pod="openshift-ingress-canary/ingress-canary-65vbf"
Sep 11 03:51:02.385782 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.385762    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfh5\" (UniqueName: \"kubernetes.io/projected/1e7b003d-b66d-4eef-8738-29ae74b53cc5-kube-api-access-rvfh5\") pod \"dns-default-5gtpq\" (UID: \"1e7b003d-b66d-4eef-8738-29ae74b53cc5\") " pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:02.385868 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.385764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb5fr\" (UniqueName: \"kubernetes.io/projected/40496107-347a-4a7d-bf59-25f0f8ac4a41-kube-api-access-xb5fr\") pod \"ingress-canary-65vbf\" (UID: \"40496107-347a-4a7d-bf59-25f0f8ac4a41\") " pod="openshift-ingress-canary/ingress-canary-65vbf"
Sep 11 03:51:02.520060 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.519963    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:02.525893 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.525877    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-65vbf"
Sep 11 03:51:02.531840 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.531767    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:02.802139 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.802115    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-65vbf"]
Sep 11 03:51:02.805611 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.805586    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5gtpq"]
Sep 11 03:51:02.805736 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:02.805665    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40496107_347a_4a7d_bf59_25f0f8ac4a41.slice/crio-db1644a757e353d115452de1b949cb66250d22c1c12e7b3d714eb3aa82a1e8f5 WatchSource:0}: Error finding container db1644a757e353d115452de1b949cb66250d22c1c12e7b3d714eb3aa82a1e8f5: Status 404 returned error can't find the container with id db1644a757e353d115452de1b949cb66250d22c1c12e7b3d714eb3aa82a1e8f5
Sep 11 03:51:02.809030 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:02.808996    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7b003d_b66d_4eef_8738_29ae74b53cc5.slice/crio-b259f97e2caa677a22aafc20e6e55d9d728d15b8251340cef3c3a0108bf06977 WatchSource:0}: Error finding container b259f97e2caa677a22aafc20e6e55d9d728d15b8251340cef3c3a0108bf06977: Status 404 returned error can't find the container with id b259f97e2caa677a22aafc20e6e55d9d728d15b8251340cef3c3a0108bf06977
Sep 11 03:51:02.855654 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.855621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:51:02.855791 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.855623    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:51:02.858174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.858155    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 03:51:02.858174 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.858155    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 03:51:02.858343 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.858159    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-58qzc\""
Sep 11 03:51:02.922855 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:02.922830    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:03.023467 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:03.023427    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:03.074753 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:03.074680    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-65vbf" event={"ID":"40496107-347a-4a7d-bf59-25f0f8ac4a41","Type":"ContainerStarted","Data":"db1644a757e353d115452de1b949cb66250d22c1c12e7b3d714eb3aa82a1e8f5"}
Sep 11 03:51:03.075800 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:03.075780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5gtpq" event={"ID":"1e7b003d-b66d-4eef-8738-29ae74b53cc5","Type":"ContainerStarted","Data":"b259f97e2caa677a22aafc20e6e55d9d728d15b8251340cef3c3a0108bf06977"}
Sep 11 03:51:03.724569 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:03.724501    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:03.781712 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:03.781672    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:03.855807 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:03.855781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:51:03.858722 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:03.858698    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 03:51:03.859230 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:03.859010    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-snbvm\""
Sep 11 03:51:03.859230 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:03.859013    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 03:51:04.139439 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.139385    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:04.207775 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.207746    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64645bfccf-ttnbj"]
Sep 11 03:51:04.210147 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.210117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.213282 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.213257    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 03:51:04.214670 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.214631    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 03:51:04.214909 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.214794    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 03:51:04.214909 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.214852    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 03:51:04.214909 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.214863    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 03:51:04.214909 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.214896    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 03:51:04.215199 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.215144    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 03:51:04.215343 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.215321    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sq84r\""
Sep 11 03:51:04.223103 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.223083    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64645bfccf-ttnbj"]
Sep 11 03:51:04.226877 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.226853    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:04.291709 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.291661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-serving-cert\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.291895 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.291759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-service-ca\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.291895 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.291822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpmct\" (UniqueName: \"kubernetes.io/projected/d7f2ad85-4d53-4814-ad2a-cb8323733682-kube-api-access-hpmct\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.291895 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.291884    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-config\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.292095 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.291930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-oauth-config\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.292095 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.291960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-oauth-serving-cert\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.294144 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.294093    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:04.393248 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.393220    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-service-ca\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.393437 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.393262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpmct\" (UniqueName: \"kubernetes.io/projected/d7f2ad85-4d53-4814-ad2a-cb8323733682-kube-api-access-hpmct\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.393437 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.393314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-config\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.393437 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.393365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-oauth-config\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.393752 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.393629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-oauth-serving-cert\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.393752 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.393670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-serving-cert\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.393945 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.393925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-service-ca\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.394096 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.394077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-config\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.394369 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.394327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-oauth-serving-cert\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.396586 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.396564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-oauth-config\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.396694 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.396673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-serving-cert\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.403042 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.402985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpmct\" (UniqueName: \"kubernetes.io/projected/d7f2ad85-4d53-4814-ad2a-cb8323733682-kube-api-access-hpmct\") pod \"console-64645bfccf-ttnbj\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") " pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:04.521078 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:04.521036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:05.083677 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:05.083641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5gtpq" event={"ID":"1e7b003d-b66d-4eef-8738-29ae74b53cc5","Type":"ContainerStarted","Data":"115757645c6ff0435941f731f1eb1bf462da0c5551d86931b58738403dbcaa1e"}
Sep 11 03:51:05.084134 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:05.083736    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5gtpq" event={"ID":"1e7b003d-b66d-4eef-8738-29ae74b53cc5","Type":"ContainerStarted","Data":"d9a66f48c9c1b032959196bbb397fa2c8d8e2214f058f9d18ca8e7596737a632"}
Sep 11 03:51:05.084134 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:05.083829    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:05.100783 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:05.100726    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5gtpq" podStartSLOduration=1.5274984900000002 podStartE2EDuration="3.100707667s" totalImagesPullingTime="1.573209177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:05.100500225 +0000 UTC m=+57.722893275" watchObservedRunningTime="2026-09-11 03:51:05.100707667 +0000 UTC m=+57.723100694"
Sep 11 03:51:05.485059 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:05.484971    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64645bfccf-ttnbj"]
Sep 11 03:51:05.528144 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:05.528100    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f2ad85_4d53_4814_ad2a_cb8323733682.slice/crio-1a7f3b11a729fa5627e8904eaf3d4803fab9901af42c34b0bc1624cadb321a1c WatchSource:0}: Error finding container 1a7f3b11a729fa5627e8904eaf3d4803fab9901af42c34b0bc1624cadb321a1c: Status 404 returned error can't find the container with id 1a7f3b11a729fa5627e8904eaf3d4803fab9901af42c34b0bc1624cadb321a1c
Sep 11 03:51:06.087630 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.087596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-65vbf" event={"ID":"40496107-347a-4a7d-bf59-25f0f8ac4a41","Type":"ContainerStarted","Data":"6b081ae24751d70b22c1351d222c7f474faa4aeb3873cb20898039ae7e57c5e6"}
Sep 11 03:51:06.088782 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.088759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64645bfccf-ttnbj" event={"ID":"d7f2ad85-4d53-4814-ad2a-cb8323733682","Type":"ContainerStarted","Data":"1a7f3b11a729fa5627e8904eaf3d4803fab9901af42c34b0bc1624cadb321a1c"}
Sep 11 03:51:06.102476 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.102436    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-65vbf" podStartSLOduration=1.346590704 podStartE2EDuration="4.102423645s" totalImagesPullingTime="2.755832941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:06.10225355 +0000 UTC m=+58.724646588" watchObservedRunningTime="2026-09-11 03:51:06.102423645 +0000 UTC m=+58.724816668"
Sep 11 03:51:06.556961 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.556908    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d655ff58d-g7k9w"]
Sep 11 03:51:06.561133 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.561101    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:06.572650 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.572630    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d655ff58d-g7k9w"]
Sep 11 03:51:06.572766 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.572727    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.580829 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.580797    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 03:51:06.610175 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.610146    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-service-ca\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.610311 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.610207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-config\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.610311 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.610238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-trusted-ca-bundle\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.610311 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.610267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-oauth-serving-cert\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.610311 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.610291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnbws\" (UniqueName: \"kubernetes.io/projected/47e759dc-b9ba-4222-8c94-4fab992a6dc8-kube-api-access-dnbws\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.610470 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.610354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-serving-cert\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.610470 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.610399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-oauth-config\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.618876 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.618846    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:06.711709 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.711674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-service-ca\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.711879 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.711726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-config\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.711879 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.711742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-trusted-ca-bundle\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.711879 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.711851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-oauth-serving-cert\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.712038 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.711880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbws\" (UniqueName: \"kubernetes.io/projected/47e759dc-b9ba-4222-8c94-4fab992a6dc8-kube-api-access-dnbws\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.712038 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.711919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-serving-cert\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.712156 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.712064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-oauth-config\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.712441 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.712417    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-service-ca\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.712660 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.712607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-trusted-ca-bundle\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.712660 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.712612    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-config\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.712660 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.712642    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-oauth-serving-cert\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.714356 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.714327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-oauth-config\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.714356 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.714352    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-serving-cert\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.720157 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.720139    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnbws\" (UniqueName: \"kubernetes.io/projected/47e759dc-b9ba-4222-8c94-4fab992a6dc8-kube-api-access-dnbws\") pod \"console-5d655ff58d-g7k9w\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") " pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:06.883073 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:06.882982    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:07.181527 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:07.181312    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d655ff58d-g7k9w"]
Sep 11 03:51:07.184077 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:07.183986    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e759dc_b9ba_4222_8c94_4fab992a6dc8.slice/crio-d0e044bfc599224dea34fe9a87b2c94f67b6439f0278c6b62fa34085a1fe40e7 WatchSource:0}: Error finding container d0e044bfc599224dea34fe9a87b2c94f67b6439f0278c6b62fa34085a1fe40e7: Status 404 returned error can't find the container with id d0e044bfc599224dea34fe9a87b2c94f67b6439f0278c6b62fa34085a1fe40e7
Sep 11 03:51:07.943197 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:07.943161    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:08.024305 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:08.024245    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:08.095087 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:08.095052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d655ff58d-g7k9w" event={"ID":"47e759dc-b9ba-4222-8c94-4fab992a6dc8","Type":"ContainerStarted","Data":"d0e044bfc599224dea34fe9a87b2c94f67b6439f0278c6b62fa34085a1fe40e7"}
Sep 11 03:51:08.123905 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:08.123873    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:08.208564 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:08.208483    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:08.307794 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:08.307755    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:08.368268 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:08.368222    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:09.098789 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:09.098752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64645bfccf-ttnbj" event={"ID":"d7f2ad85-4d53-4814-ad2a-cb8323733682","Type":"ContainerStarted","Data":"79c796a35705ef517bca3f390c6ebd48f688dfa75a725ef40b175a08911b67da"}
Sep 11 03:51:09.100264 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:09.100240    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d655ff58d-g7k9w" event={"ID":"47e759dc-b9ba-4222-8c94-4fab992a6dc8","Type":"ContainerStarted","Data":"0438c9688687b97043d52a9a4ad7730abe438b1dfb29e6d3cb400fd116169f11"}
Sep 11 03:51:09.115600 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:09.115558    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64645bfccf-ttnbj" podStartSLOduration=2.061003442 podStartE2EDuration="5.115542777s" totalImagesPullingTime="3.054539335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:09.114648304 +0000 UTC m=+61.737041332" watchObservedRunningTime="2026-09-11 03:51:09.115542777 +0000 UTC m=+61.737935804"
Sep 11 03:51:09.133992 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:09.133952    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d655ff58d-g7k9w" podStartSLOduration=1.711098523 podStartE2EDuration="3.133940537s" totalImagesPullingTime="1.422842014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:09.133291821 +0000 UTC m=+61.755684848" watchObservedRunningTime="2026-09-11 03:51:09.133940537 +0000 UTC m=+61.756333564"
Sep 11 03:51:09.218394 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:09.218359    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:09.394758 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:09.394689    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxqls"
Sep 11 03:51:09.526276 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:09.526238    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:09.628394 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:09.628362    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:09.726511 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:09.726436    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:09.884330 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:09.884296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:09.980886 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:09.980801    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:10.040640 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.040608    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn7sm"]
Sep 11 03:51:10.043540 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.043516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn7sm"
Sep 11 03:51:10.045970 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.045943    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:10.046773 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.046753    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 03:51:10.046899 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.046883    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mrdnk\""
Sep 11 03:51:10.051428 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.051401    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn7sm"]
Sep 11 03:51:10.139491 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.139453    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:10.142516 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.142482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f1ad1d47-9222-4867-9fa5-91966c1317d3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mn7sm\" (UID: \"f1ad1d47-9222-4867-9fa5-91966c1317d3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn7sm"
Sep 11 03:51:10.244137 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.244034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f1ad1d47-9222-4867-9fa5-91966c1317d3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mn7sm\" (UID: \"f1ad1d47-9222-4867-9fa5-91966c1317d3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn7sm"
Sep 11 03:51:10.246410 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.246391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f1ad1d47-9222-4867-9fa5-91966c1317d3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mn7sm\" (UID: \"f1ad1d47-9222-4867-9fa5-91966c1317d3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn7sm"
Sep 11 03:51:10.354664 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.354634    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn7sm"
Sep 11 03:51:10.381929 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.381895    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:10.481710 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.481680    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:10.621515 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.621491    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn7sm"]
Sep 11 03:51:10.623886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:10.623858    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ad1d47_9222_4867_9fa5_91966c1317d3.slice/crio-4d35c4159e354fd8f35c773035c0225b692782a095076e8decccd7bd624beedc WatchSource:0}: Error finding container 4d35c4159e354fd8f35c773035c0225b692782a095076e8decccd7bd624beedc: Status 404 returned error can't find the container with id 4d35c4159e354fd8f35c773035c0225b692782a095076e8decccd7bd624beedc
Sep 11 03:51:10.801618 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.801584    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-krd62"]
Sep 11 03:51:10.804178 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.804156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:10.806700 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.806678    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 03:51:10.806851 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.806700    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 03:51:10.806851 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.806718    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jpvnw\""
Sep 11 03:51:10.807864 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.807834    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 03:51:10.807864 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.807848    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 03:51:10.808084 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.808059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:10.823469 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.823445    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-krd62"]
Sep 11 03:51:10.848526 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.848502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:10.848616 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.848558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc29n\" (UniqueName: \"kubernetes.io/projected/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-api-access-pc29n\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:10.848616 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.848599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df2bd540-5d8f-4eaf-93da-97ad522413e0-crio-socket\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:10.848686 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.848637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:10.848686 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.848662    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df2bd540-5d8f-4eaf-93da-97ad522413e0-data-volume\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:10.873912 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.873850    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:10.949743 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.949718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:10.949840 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.949770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc29n\" (UniqueName: \"kubernetes.io/projected/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-api-access-pc29n\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:10.949840 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.949799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df2bd540-5d8f-4eaf-93da-97ad522413e0-crio-socket\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:10.949840 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.949817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:10.949840 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:10.949831    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:51:10.950016 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.949839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df2bd540-5d8f-4eaf-93da-97ad522413e0-data-volume\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:10.950016 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:10.949900    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm podName:df2bd540-5d8f-4eaf-93da-97ad522413e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:11.449884479 +0000 UTC m=+64.072277484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-krd62" (UID: "df2bd540-5d8f-4eaf-93da-97ad522413e0") : configmap "kube-rbac-proxy" not found
Sep 11 03:51:10.950016 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.949919    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df2bd540-5d8f-4eaf-93da-97ad522413e0-crio-socket\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:10.950016 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:10.949945    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:10.950016 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:10.949991    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls podName:df2bd540-5d8f-4eaf-93da-97ad522413e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:11.449979289 +0000 UTC m=+64.072372294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-krd62" (UID: "df2bd540-5d8f-4eaf-93da-97ad522413e0") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:10.950237 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.950156    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df2bd540-5d8f-4eaf-93da-97ad522413e0-data-volume\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:10.961226 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:10.961202    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc29n\" (UniqueName: \"kubernetes.io/projected/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-api-access-pc29n\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:11.105915 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:11.105884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn7sm" event={"ID":"f1ad1d47-9222-4867-9fa5-91966c1317d3","Type":"ContainerStarted","Data":"4d35c4159e354fd8f35c773035c0225b692782a095076e8decccd7bd624beedc"}
Sep 11 03:51:11.196729 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:11.196652    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:11.275383 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:11.275348    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:11.453595 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:11.453517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:11.453729 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:11.453605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:11.453729 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:11.453645    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:51:11.453729 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:11.453716    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm podName:df2bd540-5d8f-4eaf-93da-97ad522413e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:12.453696664 +0000 UTC m=+65.076089684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-krd62" (UID: "df2bd540-5d8f-4eaf-93da-97ad522413e0") : configmap "kube-rbac-proxy" not found
Sep 11 03:51:11.453729 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:11.453719    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:11.453862 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:11.453804    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls podName:df2bd540-5d8f-4eaf-93da-97ad522413e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:12.45379226 +0000 UTC m=+65.076185266 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-krd62" (UID: "df2bd540-5d8f-4eaf-93da-97ad522413e0") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:12.112123 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:12.112016    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn7sm" event={"ID":"f1ad1d47-9222-4867-9fa5-91966c1317d3","Type":"ContainerStarted","Data":"cb56cda25f760693340b035d9387b1eb3c204870e64377e6bd0985b070e973a5"}
Sep 11 03:51:12.112287 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:12.112229    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn7sm"
Sep 11 03:51:12.116934 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:12.116900    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn7sm"
Sep 11 03:51:12.127861 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:12.127819    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn7sm" podStartSLOduration=0.913264696 podStartE2EDuration="2.127804217s" totalImagesPullingTime="1.214539521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:12.127244204 +0000 UTC m=+64.749637242" watchObservedRunningTime="2026-09-11 03:51:12.127804217 +0000 UTC m=+64.750197261"
Sep 11 03:51:12.203474 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:12.203430    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:12.461206 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:12.461129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:12.461206 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:12.461190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:12.461580 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:12.461266    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:51:12.461580 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:12.461271    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:12.461580 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:12.461330    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm podName:df2bd540-5d8f-4eaf-93da-97ad522413e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:14.4613117 +0000 UTC m=+67.083704718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-krd62" (UID: "df2bd540-5d8f-4eaf-93da-97ad522413e0") : configmap "kube-rbac-proxy" not found
Sep 11 03:51:12.461580 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:12.461349    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls podName:df2bd540-5d8f-4eaf-93da-97ad522413e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:14.461338919 +0000 UTC m=+67.083731930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-krd62" (UID: "df2bd540-5d8f-4eaf-93da-97ad522413e0") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:12.598206 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:12.598175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:12.667896 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:12.667849    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:13.127496 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.127462    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:13.187588 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.187554    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"]
Sep 11 03:51:13.213323 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.213300    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"]
Sep 11 03:51:13.213438 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.213411    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"
Sep 11 03:51:13.214650 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.214619    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:13.216227 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.216209    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 03:51:13.216340 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.216293    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 03:51:13.217452 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.217430    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 03:51:13.217452 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.217443    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x68r2\""
Sep 11 03:51:13.217612 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.217464    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 03:51:13.217612 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.217472    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 03:51:13.266567 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.266542    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe31f333-a4cb-49b4-b509-05a46d249115-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-hmfjg\" (UID: \"fe31f333-a4cb-49b4-b509-05a46d249115\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"
Sep 11 03:51:13.266694 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.266575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe31f333-a4cb-49b4-b509-05a46d249115-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-hmfjg\" (UID: \"fe31f333-a4cb-49b4-b509-05a46d249115\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"
Sep 11 03:51:13.266694 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.266596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r75bw\" (UniqueName: \"kubernetes.io/projected/fe31f333-a4cb-49b4-b509-05a46d249115-kube-api-access-r75bw\") pod \"prometheus-operator-68bcd7f987-hmfjg\" (UID: \"fe31f333-a4cb-49b4-b509-05a46d249115\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"
Sep 11 03:51:13.266801 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.266721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe31f333-a4cb-49b4-b509-05a46d249115-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-hmfjg\" (UID: \"fe31f333-a4cb-49b4-b509-05a46d249115\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"
Sep 11 03:51:13.356211 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.356182    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:13.367328 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.367310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe31f333-a4cb-49b4-b509-05a46d249115-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-hmfjg\" (UID: \"fe31f333-a4cb-49b4-b509-05a46d249115\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"
Sep 11 03:51:13.367434 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.367362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe31f333-a4cb-49b4-b509-05a46d249115-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-hmfjg\" (UID: \"fe31f333-a4cb-49b4-b509-05a46d249115\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"
Sep 11 03:51:13.367434 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.367401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe31f333-a4cb-49b4-b509-05a46d249115-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-hmfjg\" (UID: \"fe31f333-a4cb-49b4-b509-05a46d249115\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"
Sep 11 03:51:13.367553 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.367441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r75bw\" (UniqueName: \"kubernetes.io/projected/fe31f333-a4cb-49b4-b509-05a46d249115-kube-api-access-r75bw\") pod \"prometheus-operator-68bcd7f987-hmfjg\" (UID: \"fe31f333-a4cb-49b4-b509-05a46d249115\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"
Sep 11 03:51:13.368304 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.368285    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe31f333-a4cb-49b4-b509-05a46d249115-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-hmfjg\" (UID: \"fe31f333-a4cb-49b4-b509-05a46d249115\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"
Sep 11 03:51:13.369855 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.369835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe31f333-a4cb-49b4-b509-05a46d249115-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-hmfjg\" (UID: \"fe31f333-a4cb-49b4-b509-05a46d249115\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"
Sep 11 03:51:13.369939 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.369844    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe31f333-a4cb-49b4-b509-05a46d249115-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-hmfjg\" (UID: \"fe31f333-a4cb-49b4-b509-05a46d249115\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"
Sep 11 03:51:13.375525 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.375502    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r75bw\" (UniqueName: \"kubernetes.io/projected/fe31f333-a4cb-49b4-b509-05a46d249115-kube-api-access-r75bw\") pod \"prometheus-operator-68bcd7f987-hmfjg\" (UID: \"fe31f333-a4cb-49b4-b509-05a46d249115\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"
Sep 11 03:51:13.456395 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.456328    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:13.522521 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.522495    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"
Sep 11 03:51:13.563878 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.563845    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:13.565012 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.564990    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d655ff58d-g7k9w_47e759dc-b9ba-4222-8c94-4fab992a6dc8/console/0.log"
Sep 11 03:51:13.668941 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.668904    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:13.669110 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.669095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs\") pod \"network-metrics-daemon-6rpfl\" (UID: \"af6ebd1b-3025-4a6f-9822-aa6606b8fa28\") " pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:51:13.669164 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.669140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:51:13.671779 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.671763    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 03:51:13.671858 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.671844    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 03:51:13.682127 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.682105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ebd1b-3025-4a6f-9822-aa6606b8fa28-metrics-certs\") pod \"network-metrics-daemon-6rpfl\" (UID: \"af6ebd1b-3025-4a6f-9822-aa6606b8fa28\") " pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:51:13.682219 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.682139    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69e1dbc6-8f47-4733-9be1-d76b49f7f4d1-original-pull-secret\") pod \"global-pull-secret-syncer-6bs2d\" (UID: \"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1\") " pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:51:13.765365 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.765295    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-64645bfccf-ttnbj_d7f2ad85-4d53-4814-ad2a-cb8323733682/console/0.log"
Sep 11 03:51:13.770289 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.770264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77rq\" (UniqueName: \"kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq\") pod \"network-check-target-t7sgk\" (UID: \"7c125f86-ef79-4811-ab32-abacb2b62402\") " pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:51:13.773305 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.773287    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 03:51:13.783260 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.783241    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 03:51:13.793808 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.793788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j77rq\" (UniqueName: \"kubernetes.io/projected/7c125f86-ef79-4811-ab32-abacb2b62402-kube-api-access-j77rq\") pod \"network-check-target-t7sgk\" (UID: \"7c125f86-ef79-4811-ab32-abacb2b62402\") " pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:51:13.810497 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.810467    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg"]
Sep 11 03:51:13.813006 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:13.812986    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe31f333_a4cb_49b4_b509_05a46d249115.slice/crio-da4005f36cf84c8ffdc3dcf3069a29c5e2a8ad94064e4d7e80866420164f10f3 WatchSource:0}: Error finding container da4005f36cf84c8ffdc3dcf3069a29c5e2a8ad94064e4d7e80866420164f10f3: Status 404 returned error can't find the container with id da4005f36cf84c8ffdc3dcf3069a29c5e2a8ad94064e4d7e80866420164f10f3
Sep 11 03:51:13.966090 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.966060    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bs2d"
Sep 11 03:51:13.972247 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.972228    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-58qzc\""
Sep 11 03:51:13.979751 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:13.979733    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6rpfl"
Sep 11 03:51:14.068977 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.068955    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-snbvm\""
Sep 11 03:51:14.077335 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.077308    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:51:14.119530 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.119497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg" event={"ID":"fe31f333-a4cb-49b4-b509-05a46d249115","Type":"ContainerStarted","Data":"da4005f36cf84c8ffdc3dcf3069a29c5e2a8ad94064e4d7e80866420164f10f3"}
Sep 11 03:51:14.178749 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.178708    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:14.257002 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.256979    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6bs2d"]
Sep 11 03:51:14.258880 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:14.258859    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e1dbc6_8f47_4733_9be1_d76b49f7f4d1.slice/crio-34573386ff444910ef4f70fb402c889931b851c3603f7c34054f50deec4e34a2 WatchSource:0}: Error finding container 34573386ff444910ef4f70fb402c889931b851c3603f7c34054f50deec4e34a2: Status 404 returned error can't find the container with id 34573386ff444910ef4f70fb402c889931b851c3603f7c34054f50deec4e34a2
Sep 11 03:51:14.260883 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.260855    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:14.281467 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.281405    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6rpfl"]
Sep 11 03:51:14.283886 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:14.283861    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf6ebd1b_3025_4a6f_9822_aa6606b8fa28.slice/crio-d63b1043df2120c2d81f8406110b83aa9e2efa1598f77deb92dd1c9096cd0003 WatchSource:0}: Error finding container d63b1043df2120c2d81f8406110b83aa9e2efa1598f77deb92dd1c9096cd0003: Status 404 returned error can't find the container with id d63b1043df2120c2d81f8406110b83aa9e2efa1598f77deb92dd1c9096cd0003
Sep 11 03:51:14.348056 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.348032    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t7sgk"]
Sep 11 03:51:14.350423 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:14.350400    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c125f86_ef79_4811_ab32_abacb2b62402.slice/crio-e31be17a7b655adc23eb2b8bc96562119731d99c8345cd90ebdfa4cd411e4436 WatchSource:0}: Error finding container e31be17a7b655adc23eb2b8bc96562119731d99c8345cd90ebdfa4cd411e4436: Status 404 returned error can't find the container with id e31be17a7b655adc23eb2b8bc96562119731d99c8345cd90ebdfa4cd411e4436
Sep 11 03:51:14.475440 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.475404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:14.475589 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.475451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:14.475589 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:14.475538    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:51:14.475697 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:14.475615    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:14.475697 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:14.475618    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm podName:df2bd540-5d8f-4eaf-93da-97ad522413e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:18.475595963 +0000 UTC m=+71.097988989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-krd62" (UID: "df2bd540-5d8f-4eaf-93da-97ad522413e0") : configmap "kube-rbac-proxy" not found
Sep 11 03:51:14.475697 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:14.475667    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls podName:df2bd540-5d8f-4eaf-93da-97ad522413e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:18.475650674 +0000 UTC m=+71.098043679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-krd62" (UID: "df2bd540-5d8f-4eaf-93da-97ad522413e0") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:14.521903 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.521868    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:14.522068 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.521933    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:14.527741 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.527718    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:14.564810 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.564783    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5gtpq_1e7b003d-b66d-4eef-8738-29ae74b53cc5/dns/0.log"
Sep 11 03:51:14.768835 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.768802    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5gtpq_1e7b003d-b66d-4eef-8738-29ae74b53cc5/kube-rbac-proxy/0.log"
Sep 11 03:51:14.898290 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.898208    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:14.998973 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:14.998928    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:15.091177 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:15.090964    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5gtpq"
Sep 11 03:51:15.126986 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:15.126951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6bs2d" event={"ID":"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1","Type":"ContainerStarted","Data":"34573386ff444910ef4f70fb402c889931b851c3603f7c34054f50deec4e34a2"}
Sep 11 03:51:15.132048 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:15.129912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6rpfl" event={"ID":"af6ebd1b-3025-4a6f-9822-aa6606b8fa28","Type":"ContainerStarted","Data":"d63b1043df2120c2d81f8406110b83aa9e2efa1598f77deb92dd1c9096cd0003"}
Sep 11 03:51:15.134435 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:15.133060    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t7sgk" event={"ID":"7c125f86-ef79-4811-ab32-abacb2b62402","Type":"ContainerStarted","Data":"e31be17a7b655adc23eb2b8bc96562119731d99c8345cd90ebdfa4cd411e4436"}
Sep 11 03:51:15.140425 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:15.140395    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:15.172881 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:15.172794    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:15.564589 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:15.564538    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4cm6g_5604fe2c-0315-490c-a234-cae850ae2d55/dns-node-resolver/0.log"
Sep 11 03:51:15.618535 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:15.618493    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:15.840817 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:15.840768    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:16.138501 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:16.138465    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg" event={"ID":"fe31f333-a4cb-49b4-b509-05a46d249115","Type":"ContainerStarted","Data":"962acc401889d5d4b5fcbb5f6df6c77dc5d8638bb665a24fae198a80b971ee3c"}
Sep 11 03:51:16.138672 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:16.138530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg" event={"ID":"fe31f333-a4cb-49b4-b509-05a46d249115","Type":"ContainerStarted","Data":"89f901d89d07b8778bc954bec72fb3f17c87f364b8fca406b636d5974133b7cb"}
Sep 11 03:51:16.140859 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:16.140829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6rpfl" event={"ID":"af6ebd1b-3025-4a6f-9822-aa6606b8fa28","Type":"ContainerStarted","Data":"770e854f7b26be602e2b9c4d32f260d69a4b9d66227d12d50a5b5884a2bda4bd"}
Sep 11 03:51:16.140967 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:16.140880    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6rpfl" event={"ID":"af6ebd1b-3025-4a6f-9822-aa6606b8fa28","Type":"ContainerStarted","Data":"9c3ff28fd9ea4dc2be8617450d6ed1087648c7756b858407bf967275602486b5"}
Sep 11 03:51:16.156284 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:16.156241    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hmfjg" podStartSLOduration=1.436920424 podStartE2EDuration="3.156222781s" totalImagesPullingTime="1.719302357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:16.1557458 +0000 UTC m=+68.778138851" watchObservedRunningTime="2026-09-11 03:51:16.156222781 +0000 UTC m=+68.778615810"
Sep 11 03:51:16.173123 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:16.173072    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6rpfl" podStartSLOduration=66.922434956 podStartE2EDuration="1m8.173054277s" totalImagesPullingTime="1.250619321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:16.172896685 +0000 UTC m=+68.795289712" watchObservedRunningTime="2026-09-11 03:51:16.173054277 +0000 UTC m=+68.795447304"
Sep 11 03:51:16.276599 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:16.276537    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:16.377156 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:16.377113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:16.763606 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:16.763572    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:16.859801 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:16.859657    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:16.883488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:16.883453    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:16.883664 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:16.883518    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:16.889357 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:16.889330    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:16.965074 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:16.964988    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mkxx2_657845f4-5ec8-4c63-9d6a-f9ca4273be0e/node-ca/0.log"
Sep 11 03:51:17.149899 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:17.149872    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:51:17.194767 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:17.194731    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64645bfccf-ttnbj"]
Sep 11 03:51:17.365236 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:17.365201    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-65vbf_40496107-347a-4a7d-bf59-25f0f8ac4a41/serve-healthcheck-canary/0.log"
Sep 11 03:51:18.290703 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.290662    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:18.513072 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.513035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:18.513262 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.513106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:18.513706 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.513683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df2bd540-5d8f-4eaf-93da-97ad522413e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:18.514387 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.514336    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:18.515819 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.515798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df2bd540-5d8f-4eaf-93da-97ad522413e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krd62\" (UID: \"df2bd540-5d8f-4eaf-93da-97ad522413e0\") " pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:18.531986 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.531959    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"]
Sep 11 03:51:18.536046 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.536006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"
Sep 11 03:51:18.538826 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.538802    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 03:51:18.540040 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.540001    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 03:51:18.540140 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.540004    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4v86k\""
Sep 11 03:51:18.548311 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.548289    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"]
Sep 11 03:51:18.549684 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.549506    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dbpcl"]
Sep 11 03:51:18.552497 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.552478    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.559639 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.559613    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 03:51:18.559639 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.559630    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 03:51:18.559861 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.559667    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 03:51:18.559938 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.559924    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k2sm5\""
Sep 11 03:51:18.613578 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.613541    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-krd62"
Sep 11 03:51:18.613742 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.613684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.613742 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.613719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e663698-55a2-49d2-9c40-136d0cf7c9be-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4qt8h\" (UID: \"7e663698-55a2-49d2-9c40-136d0cf7c9be\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"
Sep 11 03:51:18.613868 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.613740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57c8a052-e86a-4c05-bf59-d622c4962c84-sys\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.613868 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.613822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.613973 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.613869    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e663698-55a2-49d2-9c40-136d0cf7c9be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4qt8h\" (UID: \"7e663698-55a2-49d2-9c40-136d0cf7c9be\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"
Sep 11 03:51:18.613973 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.613897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-textfile\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.613973 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.613928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-tls\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.614134 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.613987    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57c8a052-e86a-4c05-bf59-d622c4962c84-root\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.614134 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.614076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e663698-55a2-49d2-9c40-136d0cf7c9be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4qt8h\" (UID: \"7e663698-55a2-49d2-9c40-136d0cf7c9be\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"
Sep 11 03:51:18.614134 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.614116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-wtmp\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.614247 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.614144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs758\" (UniqueName: \"kubernetes.io/projected/7e663698-55a2-49d2-9c40-136d0cf7c9be-kube-api-access-rs758\") pod \"openshift-state-metrics-67dcf8c979-4qt8h\" (UID: \"7e663698-55a2-49d2-9c40-136d0cf7c9be\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"
Sep 11 03:51:18.614247 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.614176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57c8a052-e86a-4c05-bf59-d622c4962c84-metrics-client-ca\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.614247 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.614205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzx9m\" (UniqueName: \"kubernetes.io/projected/57c8a052-e86a-4c05-bf59-d622c4962c84-kube-api-access-jzx9m\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.621100 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.621069    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"]
Sep 11 03:51:18.624210 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.624187    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.628033 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.627998    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 03:51:18.628294 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.628278    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 03:51:18.628665 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.628646    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 03:51:18.628763 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.628727    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cr2ks\""
Sep 11 03:51:18.656450 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.656307    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"]
Sep 11 03:51:18.666460 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.666424    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:18.715508 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3e3faf4d-01be-49e9-99e3-227ad2618ffc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.715666 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e3faf4d-01be-49e9-99e3-227ad2618ffc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.715666 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b48vh\" (UniqueName: \"kubernetes.io/projected/3e3faf4d-01be-49e9-99e3-227ad2618ffc-kube-api-access-b48vh\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.715666 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57c8a052-e86a-4c05-bf59-d622c4962c84-root\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.715831 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715664    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57c8a052-e86a-4c05-bf59-d622c4962c84-root\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.715831 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e663698-55a2-49d2-9c40-136d0cf7c9be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4qt8h\" (UID: \"7e663698-55a2-49d2-9c40-136d0cf7c9be\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"
Sep 11 03:51:18.715831 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-wtmp\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.715831 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs758\" (UniqueName: \"kubernetes.io/projected/7e663698-55a2-49d2-9c40-136d0cf7c9be-kube-api-access-rs758\") pod \"openshift-state-metrics-67dcf8c979-4qt8h\" (UID: \"7e663698-55a2-49d2-9c40-136d0cf7c9be\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"
Sep 11 03:51:18.715831 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e3faf4d-01be-49e9-99e3-227ad2618ffc-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.715831 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57c8a052-e86a-4c05-bf59-d622c4962c84-metrics-client-ca\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.715831 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzx9m\" (UniqueName: \"kubernetes.io/projected/57c8a052-e86a-4c05-bf59-d622c4962c84-kube-api-access-jzx9m\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.716251 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.716251 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3e3faf4d-01be-49e9-99e3-227ad2618ffc-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.716251 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715900    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-wtmp\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.716251 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e663698-55a2-49d2-9c40-136d0cf7c9be-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4qt8h\" (UID: \"7e663698-55a2-49d2-9c40-136d0cf7c9be\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"
Sep 11 03:51:18.716251 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57c8a052-e86a-4c05-bf59-d622c4962c84-sys\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.716251 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.715991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e3faf4d-01be-49e9-99e3-227ad2618ffc-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.716251 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.716043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.716251 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.716073    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e663698-55a2-49d2-9c40-136d0cf7c9be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4qt8h\" (UID: \"7e663698-55a2-49d2-9c40-136d0cf7c9be\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"
Sep 11 03:51:18.716251 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.716098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-textfile\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.716251 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.716130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-tls\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.716703 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.716440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57c8a052-e86a-4c05-bf59-d622c4962c84-metrics-client-ca\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.716703 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.716497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-textfile\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.716703 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.716519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57c8a052-e86a-4c05-bf59-d622c4962c84-sys\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.716703 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.716497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.716703 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:18.716600    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 03:51:18.716703 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.716619    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e663698-55a2-49d2-9c40-136d0cf7c9be-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4qt8h\" (UID: \"7e663698-55a2-49d2-9c40-136d0cf7c9be\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"
Sep 11 03:51:18.716973 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:18.716708    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-tls podName:57c8a052-e86a-4c05-bf59-d622c4962c84 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:19.21668745 +0000 UTC m=+71.839080471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-tls") pod "node-exporter-dbpcl" (UID: "57c8a052-e86a-4c05-bf59-d622c4962c84") : secret "node-exporter-tls" not found
Sep 11 03:51:18.718647 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.718626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.718755 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.718701    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e663698-55a2-49d2-9c40-136d0cf7c9be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4qt8h\" (UID: \"7e663698-55a2-49d2-9c40-136d0cf7c9be\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"
Sep 11 03:51:18.718954 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.718937    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e663698-55a2-49d2-9c40-136d0cf7c9be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4qt8h\" (UID: \"7e663698-55a2-49d2-9c40-136d0cf7c9be\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"
Sep 11 03:51:18.728494 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.728471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs758\" (UniqueName: \"kubernetes.io/projected/7e663698-55a2-49d2-9c40-136d0cf7c9be-kube-api-access-rs758\") pod \"openshift-state-metrics-67dcf8c979-4qt8h\" (UID: \"7e663698-55a2-49d2-9c40-136d0cf7c9be\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"
Sep 11 03:51:18.728789 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.728768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzx9m\" (UniqueName: \"kubernetes.io/projected/57c8a052-e86a-4c05-bf59-d622c4962c84-kube-api-access-jzx9m\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:18.747868 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.747835    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:18.817300 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.817266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3e3faf4d-01be-49e9-99e3-227ad2618ffc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.817502 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.817320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e3faf4d-01be-49e9-99e3-227ad2618ffc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.817502 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.817349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b48vh\" (UniqueName: \"kubernetes.io/projected/3e3faf4d-01be-49e9-99e3-227ad2618ffc-kube-api-access-b48vh\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.817502 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.817399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e3faf4d-01be-49e9-99e3-227ad2618ffc-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.817502 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.817437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3e3faf4d-01be-49e9-99e3-227ad2618ffc-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.817502 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.817478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e3faf4d-01be-49e9-99e3-227ad2618ffc-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.818093 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.818040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3e3faf4d-01be-49e9-99e3-227ad2618ffc-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.818239 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.818208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3e3faf4d-01be-49e9-99e3-227ad2618ffc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.818357 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.818212    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e3faf4d-01be-49e9-99e3-227ad2618ffc-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.820257 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.820234    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e3faf4d-01be-49e9-99e3-227ad2618ffc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.820590 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.820572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e3faf4d-01be-49e9-99e3-227ad2618ffc-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.827688 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.827669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b48vh\" (UniqueName: \"kubernetes.io/projected/3e3faf4d-01be-49e9-99e3-227ad2618ffc-kube-api-access-b48vh\") pod \"kube-state-metrics-59fbdd77d-c2dtj\" (UID: \"3e3faf4d-01be-49e9-99e3-227ad2618ffc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.847812 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.847785    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"
Sep 11 03:51:18.847994 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.847943    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:18.935280 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.935250    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"
Sep 11 03:51:18.948641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:18.948613    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:19.049085 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.049010    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:19.151461 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.151350    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:19.221525 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.221483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-tls\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:19.223984 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.223949    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57c8a052-e86a-4c05-bf59-d622c4962c84-node-exporter-tls\") pod \"node-exporter-dbpcl\" (UID: \"57c8a052-e86a-4c05-bf59-d622c4962c84\") " pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:19.462850 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.462703    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dbpcl"
Sep 11 03:51:19.517343 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.517296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:19.616943 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.616910    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:19.644407 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.644369    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:51:19.648770 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.648748    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.652814 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.652790    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 03:51:19.652941 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.652846    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 03:51:19.652941 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.652856    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 03:51:19.653056 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.652792    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 03:51:19.653056 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.652983    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 03:51:19.653162 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.653087    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 03:51:19.654532 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.653307    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jq4lq\""
Sep 11 03:51:19.654532 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.653437    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 03:51:19.654532 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.653485    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 03:51:19.654532 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.653665    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 03:51:19.667045 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.667000    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:51:19.715505 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.715420    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:19.718246 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.718219    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-krd62"]
Sep 11 03:51:19.719587 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.719360    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj"]
Sep 11 03:51:19.720929 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:19.720513    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e663698_55a2_49d2_9c40_136d0cf7c9be.slice/crio-3b488f2ad5e62c9bb1e445ec160b0b1967dee5962aa562699ede4acf6ea69dfc WatchSource:0}: Error finding container 3b488f2ad5e62c9bb1e445ec160b0b1967dee5962aa562699ede4acf6ea69dfc: Status 404 returned error can't find the container with id 3b488f2ad5e62c9bb1e445ec160b0b1967dee5962aa562699ede4acf6ea69dfc
Sep 11 03:51:19.720929 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.720812    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h"]
Sep 11 03:51:19.722826 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:19.722802    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2bd540_5d8f_4eaf_93da_97ad522413e0.slice/crio-08e872d569afd7e7120be2b7e750c34f646bf95377df301089d553f869efde99 WatchSource:0}: Error finding container 08e872d569afd7e7120be2b7e750c34f646bf95377df301089d553f869efde99: Status 404 returned error can't find the container with id 08e872d569afd7e7120be2b7e750c34f646bf95377df301089d553f869efde99
Sep 11 03:51:19.724141 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:19.723676    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3faf4d_01be_49e9_99e3_227ad2618ffc.slice/crio-7963f7f8463a758099a6d26e4f85c5ac0bd04a5fb10a858f0b8d64c6711bb230 WatchSource:0}: Error finding container 7963f7f8463a758099a6d26e4f85c5ac0bd04a5fb10a858f0b8d64c6711bb230: Status 404 returned error can't find the container with id 7963f7f8463a758099a6d26e4f85c5ac0bd04a5fb10a858f0b8d64c6711bb230
Sep 11 03:51:19.727508 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.727377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cb85423-2c36-4080-93d4-432b6df777e4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.727508 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.727413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.727508 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.727451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-web-config\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.727508 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.727477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7cb85423-2c36-4080-93d4-432b6df777e4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.727508 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.727504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.727708 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.727533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb85423-2c36-4080-93d4-432b6df777e4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.727708 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.727560    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-config-volume\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.727708 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.727591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7cb85423-2c36-4080-93d4-432b6df777e4-config-out\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.727708 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.727615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g9d2\" (UniqueName: \"kubernetes.io/projected/7cb85423-2c36-4080-93d4-432b6df777e4-kube-api-access-5g9d2\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.727708 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.727661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.727708 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.727690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.727928 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.727718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7cb85423-2c36-4080-93d4-432b6df777e4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.727928 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.727745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.828283 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.828206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.828283 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.828247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.828283 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.828278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7cb85423-2c36-4080-93d4-432b6df777e4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.828538 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.828309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.828538 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.828358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cb85423-2c36-4080-93d4-432b6df777e4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.828538 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.828405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.828538 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.828431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-web-config\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.828538 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.828459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7cb85423-2c36-4080-93d4-432b6df777e4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.828538 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.828494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.828538 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.828526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb85423-2c36-4080-93d4-432b6df777e4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.828876 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.828556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-config-volume\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.828876 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.828593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7cb85423-2c36-4080-93d4-432b6df777e4-config-out\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.828876 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.828599    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:19.828876 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.828618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g9d2\" (UniqueName: \"kubernetes.io/projected/7cb85423-2c36-4080-93d4-432b6df777e4-kube-api-access-5g9d2\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.829286 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.829224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7cb85423-2c36-4080-93d4-432b6df777e4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.829826 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.829708    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cb85423-2c36-4080-93d4-432b6df777e4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.831050 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.831000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb85423-2c36-4080-93d4-432b6df777e4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.831863 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.831845    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-web-config\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.832194 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.832148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.833590 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.833543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7cb85423-2c36-4080-93d4-432b6df777e4-config-out\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.833755 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.833698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7cb85423-2c36-4080-93d4-432b6df777e4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.833960 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.833896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.834146 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.834105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.834968 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.834880    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-config-volume\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.835349 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.835320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.835916 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.835897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7cb85423-2c36-4080-93d4-432b6df777e4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.837483 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.837465    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g9d2\" (UniqueName: \"kubernetes.io/projected/7cb85423-2c36-4080-93d4-432b6df777e4-kube-api-access-5g9d2\") pod \"alertmanager-main-0\" (UID: \"7cb85423-2c36-4080-93d4-432b6df777e4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:19.960525 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:19.960407    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:51:20.163590 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:20.163547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6bs2d" event={"ID":"69e1dbc6-8f47-4733-9be1-d76b49f7f4d1","Type":"ContainerStarted","Data":"8ee9baa90c2e4b6fad64c098684222a1a0242bbb7bd21c43d1de30aac2118966"}
Sep 11 03:51:20.164985 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:20.164959    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbpcl" event={"ID":"57c8a052-e86a-4c05-bf59-d622c4962c84","Type":"ContainerStarted","Data":"fed07cbdbd5f2ce9bc40775c85985e35d57e46444223912b83995a802eb0bca9"}
Sep 11 03:51:20.166699 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:20.166617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t7sgk" event={"ID":"7c125f86-ef79-4811-ab32-abacb2b62402","Type":"ContainerStarted","Data":"5295226aece1cd285066089dfb071673f9b9b50ad2f2c0bd37910e05886cbf4a"}
Sep 11 03:51:20.166797 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:20.166723    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:51:20.169076 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:20.169049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h" event={"ID":"7e663698-55a2-49d2-9c40-136d0cf7c9be","Type":"ContainerStarted","Data":"021cf5857317e87585d06cc6275399a861867c52491984c0b68123b716cd12dc"}
Sep 11 03:51:20.169183 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:20.169112    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h" event={"ID":"7e663698-55a2-49d2-9c40-136d0cf7c9be","Type":"ContainerStarted","Data":"51dcca751ca1e8fe2e7765bb7646a487c32ad98d79395e8a2d6994bdef3f6751"}
Sep 11 03:51:20.169183 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:20.169145    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h" event={"ID":"7e663698-55a2-49d2-9c40-136d0cf7c9be","Type":"ContainerStarted","Data":"3b488f2ad5e62c9bb1e445ec160b0b1967dee5962aa562699ede4acf6ea69dfc"}
Sep 11 03:51:20.170345 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:20.170324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj" event={"ID":"3e3faf4d-01be-49e9-99e3-227ad2618ffc","Type":"ContainerStarted","Data":"7963f7f8463a758099a6d26e4f85c5ac0bd04a5fb10a858f0b8d64c6711bb230"}
Sep 11 03:51:20.172238 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:20.172214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krd62" event={"ID":"df2bd540-5d8f-4eaf-93da-97ad522413e0","Type":"ContainerStarted","Data":"b593cf22917baa0d9c31bb8162518182174ed9e908a8a8739d7644f64aea30b5"}
Sep 11 03:51:20.172332 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:20.172264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krd62" event={"ID":"df2bd540-5d8f-4eaf-93da-97ad522413e0","Type":"ContainerStarted","Data":"08e872d569afd7e7120be2b7e750c34f646bf95377df301089d553f869efde99"}
Sep 11 03:51:20.180347 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:20.180299    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6bs2d" podStartSLOduration=67.018455788 podStartE2EDuration="1m12.180283439s" totalImagesPullingTime="5.161827651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:20.179110694 +0000 UTC m=+72.801503722" watchObservedRunningTime="2026-09-11 03:51:20.180283439 +0000 UTC m=+72.802676465"
Sep 11 03:51:20.195041 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:20.194961    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t7sgk" podStartSLOduration=67.108471477 podStartE2EDuration="1m12.194945347s" totalImagesPullingTime="5.08647387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:20.194040565 +0000 UTC m=+72.816433612" watchObservedRunningTime="2026-09-11 03:51:20.194945347 +0000 UTC m=+72.817338370"
Sep 11 03:51:20.247306 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:20.247163    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:51:20.363817 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:20.363782    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:20.380685 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:20.380649    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb85423_2c36_4080_93d4_432b6df777e4.slice/crio-10bb6a32d4b65bb7868fbc6c99b8cee7756d199d787b165e744f32b6730f0e91 WatchSource:0}: Error finding container 10bb6a32d4b65bb7868fbc6c99b8cee7756d199d787b165e744f32b6730f0e91: Status 404 returned error can't find the container with id 10bb6a32d4b65bb7868fbc6c99b8cee7756d199d787b165e744f32b6730f0e91
Sep 11 03:51:21.179099 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.179041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krd62" event={"ID":"df2bd540-5d8f-4eaf-93da-97ad522413e0","Type":"ContainerStarted","Data":"65833755c14e91f7948c9f3077b84c3a4db64bc801368a8b29b20e7756aab48f"}
Sep 11 03:51:21.181387 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.181351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cb85423-2c36-4080-93d4-432b6df777e4","Type":"ContainerStarted","Data":"10bb6a32d4b65bb7868fbc6c99b8cee7756d199d787b165e744f32b6730f0e91"}
Sep 11 03:51:21.183917 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.183884    2581 generic.go:356] "Generic (PLEG): container finished" podID="57c8a052-e86a-4c05-bf59-d622c4962c84" containerID="116d6b17348948cbaef696a5f6f4dccc4e02d7a80092a5134ba6a4ade60f1ee6" exitCode=0
Sep 11 03:51:21.184079 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.183931    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbpcl" event={"ID":"57c8a052-e86a-4c05-bf59-d622c4962c84","Type":"ContainerDied","Data":"116d6b17348948cbaef696a5f6f4dccc4e02d7a80092a5134ba6a4ade60f1ee6"}
Sep 11 03:51:21.513088 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.512923    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:21.537343 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.536826    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"]
Sep 11 03:51:21.541195 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.541159    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.545219 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.545051    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 03:51:21.545878 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.545435    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 03:51:21.545878 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.545478    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 03:51:21.545878 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.545642    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 03:51:21.545878 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.545718    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-skxgw\""
Sep 11 03:51:21.545878 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.545768    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bb6f1fni66mi1\""
Sep 11 03:51:21.546500 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.546371    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 03:51:21.576659 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.576630    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"]
Sep 11 03:51:21.619273 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.619241    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:21.644078 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.644045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.644262 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.644089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.644262 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.644119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.644262 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.644145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/806bef0d-627b-423d-8929-1ae0f2701a5a-metrics-client-ca\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.644262 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.644183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.644262 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.644225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-tls\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.644262 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.644243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tgcm\" (UniqueName: \"kubernetes.io/projected/806bef0d-627b-423d-8929-1ae0f2701a5a-kube-api-access-2tgcm\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.644574 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.644301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-grpc-tls\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.717188 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.717152    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:21.745242 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.745210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-tls\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.745420 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.745255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tgcm\" (UniqueName: \"kubernetes.io/projected/806bef0d-627b-423d-8929-1ae0f2701a5a-kube-api-access-2tgcm\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.745420 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.745286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-grpc-tls\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.745420 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.745333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.745420 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.745361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.745420 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.745393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.745698 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.745421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/806bef0d-627b-423d-8929-1ae0f2701a5a-metrics-client-ca\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.745698 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.745470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.751476 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.751417    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-tls\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.751926 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.751889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-grpc-tls\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.752833 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.752783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.753360 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.753302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/806bef0d-627b-423d-8929-1ae0f2701a5a-metrics-client-ca\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.755117 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.754601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.755117 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.755075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.756187 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.756135    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/806bef0d-627b-423d-8929-1ae0f2701a5a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.757097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.757075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tgcm\" (UniqueName: \"kubernetes.io/projected/806bef0d-627b-423d-8929-1ae0f2701a5a-kube-api-access-2tgcm\") pod \"thanos-querier-b4b66dbdb-q8bdv\" (UID: \"806bef0d-627b-423d-8929-1ae0f2701a5a\") " pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.817881 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.817843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:21.855589 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.855558    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:21.956691 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:21.956656    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:22.057105 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:22.057045    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:22.145089 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:22.145059    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"]
Sep 11 03:51:22.191264 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:22.191222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbpcl" event={"ID":"57c8a052-e86a-4c05-bf59-d622c4962c84","Type":"ContainerStarted","Data":"2dcc6a4fb9126f2ba8e41045e11fe32ee367efecfb98de31b8efa7cd5dd5dba3"}
Sep 11 03:51:22.191699 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:22.191298    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbpcl" event={"ID":"57c8a052-e86a-4c05-bf59-d622c4962c84","Type":"ContainerStarted","Data":"87526697352dd2b24b8948ed2e4c7349cdce5ed19c2e9c44323820bbf4970967"}
Sep 11 03:51:22.193670 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:22.193643    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h" event={"ID":"7e663698-55a2-49d2-9c40-136d0cf7c9be","Type":"ContainerStarted","Data":"145a59b8061777d50150a2ce22b3a656fff8c6a8dcdc1689bece0c6a1c05c58b"}
Sep 11 03:51:22.196073 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:22.196047    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj" event={"ID":"3e3faf4d-01be-49e9-99e3-227ad2618ffc","Type":"ContainerStarted","Data":"bb75d31d69bf641941d0e0e90619bd7bc81e1fea7fcad1f23271f55d58a10b13"}
Sep 11 03:51:22.196151 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:22.196095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj" event={"ID":"3e3faf4d-01be-49e9-99e3-227ad2618ffc","Type":"ContainerStarted","Data":"5c7b04ccf277d61143115a65464112ca3770837a6e77f183304da17565f07d5f"}
Sep 11 03:51:22.196151 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:22.196123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj" event={"ID":"3e3faf4d-01be-49e9-99e3-227ad2618ffc","Type":"ContainerStarted","Data":"1db55a83d982dbdf8860e46d25122c2bf776a6c04589b81d294f48d7cd9e5037"}
Sep 11 03:51:22.217809 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:22.217743    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dbpcl" podStartSLOduration=3.113023558 podStartE2EDuration="4.217724232s" totalImagesPullingTime="1.02964648s" totalInitContainerRuntime="75.054194ms" isStatefulPod=true podCreationTimestamp="2026-09-11 03:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:22.217690657 +0000 UTC m=+74.840083695" watchObservedRunningTime="2026-09-11 03:51:22.217724232 +0000 UTC m=+74.840117256"
Sep 11 03:51:22.244536 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:22.244478    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4qt8h" podStartSLOduration=2.875322953 podStartE2EDuration="4.244459724s" totalImagesPullingTime="1.369136771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:22.243353687 +0000 UTC m=+74.865746736" watchObservedRunningTime="2026-09-11 03:51:22.244459724 +0000 UTC m=+74.866852753"
Sep 11 03:51:22.266755 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:22.266349    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-c2dtj" podStartSLOduration=2.750870805 podStartE2EDuration="4.266329452s" totalImagesPullingTime="1.515458647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:22.266102017 +0000 UTC m=+74.888495068" watchObservedRunningTime="2026-09-11 03:51:22.266329452 +0000 UTC m=+74.888722479"
Sep 11 03:51:22.289248 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:22.289219    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806bef0d_627b_423d_8929_1ae0f2701a5a.slice/crio-5c727b06e5f74d4274e4d594fb9053008b90db9e896cfa575f55ee647cc326ec WatchSource:0}: Error finding container 5c727b06e5f74d4274e4d594fb9053008b90db9e896cfa575f55ee647cc326ec: Status 404 returned error can't find the container with id 5c727b06e5f74d4274e4d594fb9053008b90db9e896cfa575f55ee647cc326ec
Sep 11 03:51:23.040246 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.038893    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-bb4598894-mcwhb"]
Sep 11 03:51:23.042528 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.042498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.045264 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.045231    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 03:51:23.045362 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.045265    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 03:51:23.046464 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.046445    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 03:51:23.046464 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.046457    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 03:51:23.046636 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.046542    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fm699\""
Sep 11 03:51:23.046636 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.046620    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9q0704uc1fj1o\""
Sep 11 03:51:23.050718 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.050680    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bb4598894-mcwhb"]
Sep 11 03:51:23.158277 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.158245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5fcb49d-518f-4960-9ab6-33155afc702b-audit-log\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.158452 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.158289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5fcb49d-518f-4960-9ab6-33155afc702b-metrics-server-audit-profiles\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.158452 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.158363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmmr2\" (UniqueName: \"kubernetes.io/projected/d5fcb49d-518f-4960-9ab6-33155afc702b-kube-api-access-zmmr2\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.158452 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.158409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5fcb49d-518f-4960-9ab6-33155afc702b-secret-metrics-server-tls\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.158618 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.158477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5fcb49d-518f-4960-9ab6-33155afc702b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.158618 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.158513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5fcb49d-518f-4960-9ab6-33155afc702b-client-ca-bundle\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.158618 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.158593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5fcb49d-518f-4960-9ab6-33155afc702b-secret-metrics-server-client-certs\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.201344 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.201303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv" event={"ID":"806bef0d-627b-423d-8929-1ae0f2701a5a","Type":"ContainerStarted","Data":"5c727b06e5f74d4274e4d594fb9053008b90db9e896cfa575f55ee647cc326ec"}
Sep 11 03:51:23.203761 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.203734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krd62" event={"ID":"df2bd540-5d8f-4eaf-93da-97ad522413e0","Type":"ContainerStarted","Data":"ac8db309d57f8c1b1bd5ca05f9965b94d06ed98bca09be8bdb230b5c5352df74"}
Sep 11 03:51:23.205481 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.205458    2581 generic.go:356] "Generic (PLEG): container finished" podID="7cb85423-2c36-4080-93d4-432b6df777e4" containerID="d6751dadb4120462cb28d40443c37126722fc25cbe702af889ec10888368369d" exitCode=0
Sep 11 03:51:23.205598 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.205572    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cb85423-2c36-4080-93d4-432b6df777e4","Type":"ContainerDied","Data":"d6751dadb4120462cb28d40443c37126722fc25cbe702af889ec10888368369d"}
Sep 11 03:51:23.223550 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.223509    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-krd62" podStartSLOduration=10.762843841 podStartE2EDuration="13.223492399s" totalImagesPullingTime="2.460648558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:51:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:23.223217756 +0000 UTC m=+75.845610811" watchObservedRunningTime="2026-09-11 03:51:23.223492399 +0000 UTC m=+75.845885470"
Sep 11 03:51:23.259830 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.259798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5fcb49d-518f-4960-9ab6-33155afc702b-secret-metrics-server-client-certs\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.260501 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.260302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5fcb49d-518f-4960-9ab6-33155afc702b-audit-log\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.260501 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.260477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5fcb49d-518f-4960-9ab6-33155afc702b-metrics-server-audit-profiles\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.260688 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.260556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmmr2\" (UniqueName: \"kubernetes.io/projected/d5fcb49d-518f-4960-9ab6-33155afc702b-kube-api-access-zmmr2\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.260688 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.260633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5fcb49d-518f-4960-9ab6-33155afc702b-secret-metrics-server-tls\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.260788 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.260691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5fcb49d-518f-4960-9ab6-33155afc702b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.260788 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.260754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5fcb49d-518f-4960-9ab6-33155afc702b-client-ca-bundle\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.261854 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.260899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5fcb49d-518f-4960-9ab6-33155afc702b-audit-log\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.261854 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.261464    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5fcb49d-518f-4960-9ab6-33155afc702b-metrics-server-audit-profiles\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.262047 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.261989    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5fcb49d-518f-4960-9ab6-33155afc702b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.263234 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.263212    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5fcb49d-518f-4960-9ab6-33155afc702b-secret-metrics-server-client-certs\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.263841 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.263819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5fcb49d-518f-4960-9ab6-33155afc702b-client-ca-bundle\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.264172 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.264151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5fcb49d-518f-4960-9ab6-33155afc702b-secret-metrics-server-tls\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.273623 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.273595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmmr2\" (UniqueName: \"kubernetes.io/projected/d5fcb49d-518f-4960-9ab6-33155afc702b-kube-api-access-zmmr2\") pod \"metrics-server-bb4598894-mcwhb\" (UID: \"d5fcb49d-518f-4960-9ab6-33155afc702b\") " pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.312958 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.312888    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:23.334106 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.334083    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv"]
Sep 11 03:51:23.339316 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.339292    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f668854cb-zg2gn"]
Sep 11 03:51:23.339460 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.339430    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv"
Sep 11 03:51:23.341752 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.341733    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 03:51:23.341863 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.341775    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7v76g\""
Sep 11 03:51:23.343425 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.343409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.345417 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.345396    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv"]
Sep 11 03:51:23.348331 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.348313    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f668854cb-zg2gn"]
Sep 11 03:51:23.354233 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.354215    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:23.385522 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.385460    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:23.453269 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.453184    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:23.463166 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.463132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn2xt\" (UniqueName: \"kubernetes.io/projected/d582a9c9-63bd-48cd-a884-d83c49989227-kube-api-access-zn2xt\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.463291 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.463194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d582a9c9-63bd-48cd-a884-d83c49989227-console-oauth-config\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.463291 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.463246    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d582a9c9-63bd-48cd-a884-d83c49989227-console-serving-cert\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.463291 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.463275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-console-config\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.463452 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.463345    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa49c5a7-80aa-4a35-b4fe-30a674060be7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-6qnsv\" (UID: \"fa49c5a7-80aa-4a35-b4fe-30a674060be7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv"
Sep 11 03:51:23.463452 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.463422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-service-ca\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.463556 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.463461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-oauth-serving-cert\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.463556 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.463499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-trusted-ca-bundle\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.463556 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.463522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p7h4\" (UniqueName: \"kubernetes.io/projected/fa49c5a7-80aa-4a35-b4fe-30a674060be7-kube-api-access-4p7h4\") pod \"monitoring-plugin-5d85778b9d-6qnsv\" (UID: \"fa49c5a7-80aa-4a35-b4fe-30a674060be7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv"
Sep 11 03:51:23.564484 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.564395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn2xt\" (UniqueName: \"kubernetes.io/projected/d582a9c9-63bd-48cd-a884-d83c49989227-kube-api-access-zn2xt\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.564484 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.564467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d582a9c9-63bd-48cd-a884-d83c49989227-console-oauth-config\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.564704 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.564498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d582a9c9-63bd-48cd-a884-d83c49989227-console-serving-cert\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.564704 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.564529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-console-config\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.564704 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.564559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa49c5a7-80aa-4a35-b4fe-30a674060be7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-6qnsv\" (UID: \"fa49c5a7-80aa-4a35-b4fe-30a674060be7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv"
Sep 11 03:51:23.564704 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.564593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-service-ca\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.564704 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.564614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-oauth-serving-cert\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.564704 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.564648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-trusted-ca-bundle\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.564704 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.564664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p7h4\" (UniqueName: \"kubernetes.io/projected/fa49c5a7-80aa-4a35-b4fe-30a674060be7-kube-api-access-4p7h4\") pod \"monitoring-plugin-5d85778b9d-6qnsv\" (UID: \"fa49c5a7-80aa-4a35-b4fe-30a674060be7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv"
Sep 11 03:51:23.565746 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.565406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-console-config\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.565746 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.565460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-service-ca\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.565746 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.565677    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-oauth-serving-cert\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.566220 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.566189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-trusted-ca-bundle\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.567204 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.567180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d582a9c9-63bd-48cd-a884-d83c49989227-console-oauth-config\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.567485 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.567436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d582a9c9-63bd-48cd-a884-d83c49989227-console-serving-cert\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.567866 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.567847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa49c5a7-80aa-4a35-b4fe-30a674060be7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-6qnsv\" (UID: \"fa49c5a7-80aa-4a35-b4fe-30a674060be7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv"
Sep 11 03:51:23.572469 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.572442    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn2xt\" (UniqueName: \"kubernetes.io/projected/d582a9c9-63bd-48cd-a884-d83c49989227-kube-api-access-zn2xt\") pod \"console-6f668854cb-zg2gn\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") " pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.573072 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.573051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p7h4\" (UniqueName: \"kubernetes.io/projected/fa49c5a7-80aa-4a35-b4fe-30a674060be7-kube-api-access-4p7h4\") pod \"monitoring-plugin-5d85778b9d-6qnsv\" (UID: \"fa49c5a7-80aa-4a35-b4fe-30a674060be7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv"
Sep 11 03:51:23.636647 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.636620    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bb4598894-mcwhb"]
Sep 11 03:51:23.655208 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.655179    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv"
Sep 11 03:51:23.663440 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.663402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:23.865422 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.865101    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7fc8677df8-cplcz"]
Sep 11 03:51:23.869269 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.869245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:23.874510 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.874410    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 03:51:23.874634 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.874602    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 03:51:23.874634 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.874625    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 03:51:23.874751 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.874628    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 03:51:23.874937 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.874914    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 03:51:23.875034 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.874993    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4rdsq\""
Sep 11 03:51:23.875998 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.875978    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fc8677df8-cplcz"]
Sep 11 03:51:23.879822 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.879797    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 03:51:23.968461 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.968421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-secret-telemeter-client\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:23.968649 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.968477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-metrics-client-ca\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:23.968649 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.968526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:23.968649 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.968554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-telemeter-client-tls\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:23.968649 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.968593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-federate-client-tls\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:23.968875 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.968700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:23.968875 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.968739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-serving-certs-ca-bundle\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:23.968875 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:23.968766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhtmm\" (UniqueName: \"kubernetes.io/projected/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-kube-api-access-fhtmm\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.070055 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.070009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-metrics-client-ca\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.070225 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.070082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.070225 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.070118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-telemeter-client-tls\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.070225 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.070158    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-federate-client-tls\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.070225 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.070198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.070440 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.070238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-serving-certs-ca-bundle\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.070440 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.070271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhtmm\" (UniqueName: \"kubernetes.io/projected/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-kube-api-access-fhtmm\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.070440 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.070372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-secret-telemeter-client\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.070953 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.070897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-metrics-client-ca\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.071328 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.071303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-serving-certs-ca-bundle\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.071457 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.071420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.073136 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.073113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-federate-client-tls\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.073252 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.073199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.073491 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.073467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-secret-telemeter-client\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.073587 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.073467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-telemeter-client-tls\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.078976 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.078952    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhtmm\" (UniqueName: \"kubernetes.io/projected/01dec1b6-1add-4f76-8e6b-eb91561ebbc8-kube-api-access-fhtmm\") pod \"telemeter-client-7fc8677df8-cplcz\" (UID: \"01dec1b6-1add-4f76-8e6b-eb91561ebbc8\") " pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.182789 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.182703    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz"
Sep 11 03:51:24.208346 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:24.208317    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fcb49d_518f_4960_9ab6_33155afc702b.slice/crio-674f752756ee0ca8e19b4e8e6d576a7e7b7cf2ae386d82d2d9cca3f6180ded28 WatchSource:0}: Error finding container 674f752756ee0ca8e19b4e8e6d576a7e7b7cf2ae386d82d2d9cca3f6180ded28: Status 404 returned error can't find the container with id 674f752756ee0ca8e19b4e8e6d576a7e7b7cf2ae386d82d2d9cca3f6180ded28
Sep 11 03:51:24.228815 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.228786    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:24.330408 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.330365    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:24.636557 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.636525    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:24.743863 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.743744    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:24.831648 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.831621    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:51:24.837052 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.836530    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.840465 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.839787    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 03:51:24.840465 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.839998    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6e9oualjg8sp0\""
Sep 11 03:51:24.840465 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.840289    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 03:51:24.840765 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.840745    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 03:51:24.840765 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.840759    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-m5h57\""
Sep 11 03:51:24.841125 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.840973    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 03:51:24.841125 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.841045    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 03:51:24.841258 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.841200    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 03:51:24.841258 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.841227    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 03:51:24.841366 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.841276    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 03:51:24.841366 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.841322    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 03:51:24.841366 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.841337    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 03:51:24.841366 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.841231    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 03:51:24.841635 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.841284    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 03:51:24.845010 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.844990    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 03:51:24.860895 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.860869    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 03:51:24.866805 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.866756    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:24.866943 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.866911    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:51:24.944796 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.944729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv"]
Sep 11 03:51:24.950817 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.950793    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f668854cb-zg2gn"]
Sep 11 03:51:24.958012 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.957944    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fc8677df8-cplcz"]
Sep 11 03:51:24.977836 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.977807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78d1ed64-d42b-4b80-a0ec-68f904e00b11-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.977942 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.977849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.977942 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.977872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.977942 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.977908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978126 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.977947    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978126 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.977979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978126 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.978002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978126 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.978043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978126 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.978069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78d1ed64-d42b-4b80-a0ec-68f904e00b11-config-out\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978126 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.978095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978126 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.978119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978449 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.978222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978449 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.978251    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978449 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.978282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978449 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.978323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978449 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.978345    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-web-config\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978449 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.978378    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978449 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.978403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8hg8\" (UniqueName: \"kubernetes.io/projected/78d1ed64-d42b-4b80-a0ec-68f904e00b11-kube-api-access-g8hg8\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978449 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.978434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-config\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:24.978919 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:24.978463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.079579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.079634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.079665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.079691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.079720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.079747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78d1ed64-d42b-4b80-a0ec-68f904e00b11-config-out\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.079771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.079797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.079843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.079872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.079907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.079949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.079972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-web-config\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.080005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080051 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.080045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hg8\" (UniqueName: \"kubernetes.io/projected/78d1ed64-d42b-4b80-a0ec-68f904e00b11-kube-api-access-g8hg8\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080863 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.080081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-config\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080863 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.080115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080863 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.080149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78d1ed64-d42b-4b80-a0ec-68f904e00b11-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080863 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.080172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.080863 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.080197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.081769 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.081275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.081769 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.081462    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.082989 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.082529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.083617 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.083296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.085851 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.084088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-config\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.086738 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.086477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.086814 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.086764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.087994 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.087464    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.087994 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.087526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.087994 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.087912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.087994 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.087950    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.088556 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.088522    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78d1ed64-d42b-4b80-a0ec-68f904e00b11-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.090780 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.090591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78d1ed64-d42b-4b80-a0ec-68f904e00b11-config-out\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.090780 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.090753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78d1ed64-d42b-4b80-a0ec-68f904e00b11-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.091749 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.091709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.091892 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.091868    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.092743 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.092696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-web-config\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.093702 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.093623    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.096428 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.096378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78d1ed64-d42b-4b80-a0ec-68f904e00b11-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.096895 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.096873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hg8\" (UniqueName: \"kubernetes.io/projected/78d1ed64-d42b-4b80-a0ec-68f904e00b11-kube-api-access-g8hg8\") pod \"prometheus-k8s-0\" (UID: \"78d1ed64-d42b-4b80-a0ec-68f904e00b11\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.167006 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.166875    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:25.223126 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.223084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv" event={"ID":"806bef0d-627b-423d-8929-1ae0f2701a5a","Type":"ContainerStarted","Data":"554709c52a7b3b2eb3b39abeee2e7647e49e58ccfaa6706c1e6e259f667fa9d2"}
Sep 11 03:51:25.223588 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.223178    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv" event={"ID":"806bef0d-627b-423d-8929-1ae0f2701a5a","Type":"ContainerStarted","Data":"e17ca1e1aee9a9c6458f89c8fb8f21667e549c4dfec44fc6ea2fd0c3c1a0e85a"}
Sep 11 03:51:25.223588 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.223230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv" event={"ID":"806bef0d-627b-423d-8929-1ae0f2701a5a","Type":"ContainerStarted","Data":"6a57383fc1d6e7ef58cb274aca15d0845a67a67fb1357a6b336da3d9aa0b7e9a"}
Sep 11 03:51:25.225370 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.225336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f668854cb-zg2gn" event={"ID":"d582a9c9-63bd-48cd-a884-d83c49989227","Type":"ContainerStarted","Data":"aa9137539fd252b464699014ddaa312b703fb993918345b58c0060fd032f5d04"}
Sep 11 03:51:25.225457 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.225395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f668854cb-zg2gn" event={"ID":"d582a9c9-63bd-48cd-a884-d83c49989227","Type":"ContainerStarted","Data":"b73c8594ece8897bfff80f62bd0f7cecdbbf38311f82241970d4722cab49245d"}
Sep 11 03:51:25.232591 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.232138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cb85423-2c36-4080-93d4-432b6df777e4","Type":"ContainerStarted","Data":"6600ae180629a0687371932e816fbf875e314a56a521aca3360f7b5abae52524"}
Sep 11 03:51:25.232591 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.232232    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cb85423-2c36-4080-93d4-432b6df777e4","Type":"ContainerStarted","Data":"82a9c974a1fb2ff2031351095db49491a6133a074b2507b769f7ad820c52326a"}
Sep 11 03:51:25.232591 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.232434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cb85423-2c36-4080-93d4-432b6df777e4","Type":"ContainerStarted","Data":"8566b3a8a4780208dae13c6cad2666ca6d4665656b0e55405664af5fec08a4d9"}
Sep 11 03:51:25.232591 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.232494    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cb85423-2c36-4080-93d4-432b6df777e4","Type":"ContainerStarted","Data":"14f32fe5c7362c34114c6b0877448af4b8ed7e5084de6354b53fbb5717b0ad67"}
Sep 11 03:51:25.232591 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.232542    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cb85423-2c36-4080-93d4-432b6df777e4","Type":"ContainerStarted","Data":"f6110fe353c720fa29306ff211fc65c8ebc8b985eb9dc734378c66896fb86f0d"}
Sep 11 03:51:25.234598 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.234493    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:25.236103 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.236079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz" event={"ID":"01dec1b6-1add-4f76-8e6b-eb91561ebbc8","Type":"ContainerStarted","Data":"d3f11c6d8fe7096e88d05c5e9a9ff6f8ba8de6cffcbaeb4fbcc2510e033a9aa5"}
Sep 11 03:51:25.237665 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.237643    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv" event={"ID":"fa49c5a7-80aa-4a35-b4fe-30a674060be7","Type":"ContainerStarted","Data":"f79ffbcb456b2d2846e111acfca3d20ae71be6a2c0bc2976623a5f8776032330"}
Sep 11 03:51:25.241893 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.241612    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bb4598894-mcwhb" event={"ID":"d5fcb49d-518f-4960-9ab6-33155afc702b","Type":"ContainerStarted","Data":"674f752756ee0ca8e19b4e8e6d576a7e7b7cf2ae386d82d2d9cca3f6180ded28"}
Sep 11 03:51:25.244042 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.243760    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f668854cb-zg2gn" podStartSLOduration=2.243743746 podStartE2EDuration="2.243743746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:25.243199201 +0000 UTC m=+77.865592250" watchObservedRunningTime="2026-09-11 03:51:25.243743746 +0000 UTC m=+77.866136774"
Sep 11 03:51:25.332586 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.332543    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:25.393395 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.393357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:25.464357 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:25.464325    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:51:25.466368 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:51:25.466340    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d1ed64_d42b_4b80_a0ec_68f904e00b11.slice/crio-346128f80dde0eccd61a098f3bf86a08b1987c1589b73c66c52b40d20cc17cac WatchSource:0}: Error finding container 346128f80dde0eccd61a098f3bf86a08b1987c1589b73c66c52b40d20cc17cac: Status 404 returned error can't find the container with id 346128f80dde0eccd61a098f3bf86a08b1987c1589b73c66c52b40d20cc17cac
Sep 11 03:51:26.253370 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:26.252398    2581 generic.go:356] "Generic (PLEG): container finished" podID="78d1ed64-d42b-4b80-a0ec-68f904e00b11" containerID="78e8b7d43716c439a8e2e27f71ae40e2af3cd0f63b9fbe79d5e312c31cf6f261" exitCode=0
Sep 11 03:51:26.253370 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:26.252548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78d1ed64-d42b-4b80-a0ec-68f904e00b11","Type":"ContainerDied","Data":"78e8b7d43716c439a8e2e27f71ae40e2af3cd0f63b9fbe79d5e312c31cf6f261"}
Sep 11 03:51:26.253370 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:26.252623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78d1ed64-d42b-4b80-a0ec-68f904e00b11","Type":"ContainerStarted","Data":"346128f80dde0eccd61a098f3bf86a08b1987c1589b73c66c52b40d20cc17cac"}
Sep 11 03:51:26.263791 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:26.262408    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cb85423-2c36-4080-93d4-432b6df777e4","Type":"ContainerStarted","Data":"f9bd1e6fb65700170de7670ae58253275de960092ea843a660585708b1c85ac4"}
Sep 11 03:51:26.271673 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:26.271188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bb4598894-mcwhb" event={"ID":"d5fcb49d-518f-4960-9ab6-33155afc702b","Type":"ContainerStarted","Data":"e40191d035a7cb3612ef897b3ed87356115929c9ac9b08853c61dd62cee58dac"}
Sep 11 03:51:26.276009 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:26.275918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv" event={"ID":"806bef0d-627b-423d-8929-1ae0f2701a5a","Type":"ContainerStarted","Data":"cb08caffd999208e9a7f6c0987405c0c42aca38d9bd38029d57e688b6b18b868"}
Sep 11 03:51:26.306742 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:26.306692    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-bb4598894-mcwhb" podStartSLOduration=1.423436985 podStartE2EDuration="3.306672421s" totalImagesPullingTime="1.883235436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:26.305930816 +0000 UTC m=+78.928323879" watchObservedRunningTime="2026-09-11 03:51:26.306672421 +0000 UTC m=+78.929065451"
Sep 11 03:51:26.336101 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:26.334818    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.980692258 podStartE2EDuration="7.33479547s" totalImagesPullingTime="4.338598696s" totalInitContainerRuntime="15.504516ms" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:26.332428272 +0000 UTC m=+78.954821316" watchObservedRunningTime="2026-09-11 03:51:26.33479547 +0000 UTC m=+78.957188498"
Sep 11 03:51:27.284606 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:27.284569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv" event={"ID":"806bef0d-627b-423d-8929-1ae0f2701a5a","Type":"ContainerStarted","Data":"417f3c9124092d73dfc59c7d689188d89ae9c8ca671da8b5b8f5e5073f2ebc3a"}
Sep 11 03:51:27.285091 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:27.284652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv" event={"ID":"806bef0d-627b-423d-8929-1ae0f2701a5a","Type":"ContainerStarted","Data":"9cf23bc47cfa25f1fdf58f8f58a845fdc69796e30defb70c2bc5d772095d4a64"}
Sep 11 03:51:27.308154 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:27.308098    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv" podStartSLOduration=2.785826114 podStartE2EDuration="6.308074691s" totalImagesPullingTime="3.522248577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:27.307171797 +0000 UTC m=+79.929564845" watchObservedRunningTime="2026-09-11 03:51:27.308074691 +0000 UTC m=+79.930467719"
Sep 11 03:51:28.291813 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:28.291773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz" event={"ID":"01dec1b6-1add-4f76-8e6b-eb91561ebbc8","Type":"ContainerStarted","Data":"38165b2c9d539f3e411d01440b06832c2cbf55fdfff266816833133ee61ff2c5"}
Sep 11 03:51:28.292321 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:28.291841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz" event={"ID":"01dec1b6-1add-4f76-8e6b-eb91561ebbc8","Type":"ContainerStarted","Data":"4282682f7736d73f850a7511e08177be0104288908c56d9f161f62081ff774fd"}
Sep 11 03:51:28.292321 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:28.291873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz" event={"ID":"01dec1b6-1add-4f76-8e6b-eb91561ebbc8","Type":"ContainerStarted","Data":"9b87142b7ac2c43f0051fbc30af7e080d4378869bd8103cc0df9188ad8a545be"}
Sep 11 03:51:28.293520 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:28.293489    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv" event={"ID":"fa49c5a7-80aa-4a35-b4fe-30a674060be7","Type":"ContainerStarted","Data":"647a34bd2b8a5fd6e6b1b14653ebabf40d0aac0c739e31472fb1dcee2f707418"}
Sep 11 03:51:28.293642 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:28.293544    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv"
Sep 11 03:51:28.293838 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:28.293818    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:28.298667 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:28.298648    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv"
Sep 11 03:51:28.316637 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:28.316589    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7fc8677df8-cplcz" podStartSLOduration=2.748861441 podStartE2EDuration="5.316575444s" totalImagesPullingTime="2.567714003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:28.314708926 +0000 UTC m=+80.937101967" watchObservedRunningTime="2026-09-11 03:51:28.316575444 +0000 UTC m=+80.938968471"
Sep 11 03:51:28.332968 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:28.332932    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6qnsv" podStartSLOduration=2.759100696 podStartE2EDuration="5.332921085s" totalImagesPullingTime="2.573820389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:28.332513159 +0000 UTC m=+80.954906208" watchObservedRunningTime="2026-09-11 03:51:28.332921085 +0000 UTC m=+80.955314112"
Sep 11 03:51:28.421380 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:28.421349    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f668854cb-zg2gn"]
Sep 11 03:51:29.316780 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:29.316751    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b4b66dbdb-q8bdv"
Sep 11 03:51:29.903589 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:29.903552    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:29.984774 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:29.984698    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:30.312457 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:30.312426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78d1ed64-d42b-4b80-a0ec-68f904e00b11","Type":"ContainerStarted","Data":"bf937de440c7fc8555de144b881c5628a10ef6604244846dece2992fa01cb20e"}
Sep 11 03:51:31.320029 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:31.319999    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78d1ed64-d42b-4b80-a0ec-68f904e00b11","Type":"ContainerStarted","Data":"a9c2d7d4236d0f6ec0340121d1e0f06339cfb68dec451b3c3ba33442bcac11f5"}
Sep 11 03:51:31.320414 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:31.320081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78d1ed64-d42b-4b80-a0ec-68f904e00b11","Type":"ContainerStarted","Data":"9f1e54153a34c299bfa598d061c088c39e3356dde0b47d9e84cbfb5b542ac0cf"}
Sep 11 03:51:31.320414 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:31.320108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78d1ed64-d42b-4b80-a0ec-68f904e00b11","Type":"ContainerStarted","Data":"07b3f312233fada669d054c70f6b06d4c9672490c38bdde7ed62b0213ab95baf"}
Sep 11 03:51:31.320414 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:31.320133    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78d1ed64-d42b-4b80-a0ec-68f904e00b11","Type":"ContainerStarted","Data":"e79037596545ffc4f813e692231d9bf94d59d699e7889d2552b3759ae8bcfd7d"}
Sep 11 03:51:31.320414 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:31.320159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78d1ed64-d42b-4b80-a0ec-68f904e00b11","Type":"ContainerStarted","Data":"e27c91934e32ee6a6aa07f5a80b64dc9c8f6ecaf7172be9fbd402d27faae493c"}
Sep 11 03:51:31.350252 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:31.350203    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.361793128 podStartE2EDuration="7.350187275s" totalImagesPullingTime="3.95870452s" totalInitContainerRuntime="29.689627ms" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:31.34771997 +0000 UTC m=+83.970113028" watchObservedRunningTime="2026-09-11 03:51:31.350187275 +0000 UTC m=+83.972580301"
Sep 11 03:51:33.266386 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:33.266345    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:33.365770 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:33.365735    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:33.664066 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:33.664034    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:34.143093 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:34.143050    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:34.197755 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:34.197721    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:35.167263 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:35.167235    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:51:36.614947 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:36.614908    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:39.328806 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:39.328776    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d655ff58d-g7k9w"]
Sep 11 03:51:39.428485 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:39.428453    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:39.481226 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:39.481198    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:39.543596 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:39.543564    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:42.218562 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.218519    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64645bfccf-ttnbj" podUID="d7f2ad85-4d53-4814-ad2a-cb8323733682" containerName="console" containerID="cri-o://79c796a35705ef517bca3f390c6ebd48f688dfa75a725ef40b175a08911b67da" gracePeriod=15
Sep 11 03:51:42.368099 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.368066    2581 generic.go:356] "Generic (PLEG): container finished" podID="d7f2ad85-4d53-4814-ad2a-cb8323733682" containerID="79c796a35705ef517bca3f390c6ebd48f688dfa75a725ef40b175a08911b67da" exitCode=0
Sep 11 03:51:42.368233 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.368143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64645bfccf-ttnbj" event={"ID":"d7f2ad85-4d53-4814-ad2a-cb8323733682","Type":"ContainerDied","Data":"79c796a35705ef517bca3f390c6ebd48f688dfa75a725ef40b175a08911b67da"}
Sep 11 03:51:42.483662 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.483602    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:42.637136 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.637105    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-config\") pod \"d7f2ad85-4d53-4814-ad2a-cb8323733682\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") "
Sep 11 03:51:42.637298 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.637156    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-oauth-config\") pod \"d7f2ad85-4d53-4814-ad2a-cb8323733682\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") "
Sep 11 03:51:42.637298 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.637187    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-serving-cert\") pod \"d7f2ad85-4d53-4814-ad2a-cb8323733682\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") "
Sep 11 03:51:42.637298 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.637249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpmct\" (UniqueName: \"kubernetes.io/projected/d7f2ad85-4d53-4814-ad2a-cb8323733682-kube-api-access-hpmct\") pod \"d7f2ad85-4d53-4814-ad2a-cb8323733682\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") "
Sep 11 03:51:42.637298 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.637294    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-service-ca\") pod \"d7f2ad85-4d53-4814-ad2a-cb8323733682\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") "
Sep 11 03:51:42.637502 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.637314    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-oauth-serving-cert\") pod \"d7f2ad85-4d53-4814-ad2a-cb8323733682\" (UID: \"d7f2ad85-4d53-4814-ad2a-cb8323733682\") "
Sep 11 03:51:42.637557 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.637522    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-config" (OuterVolumeSpecName: "console-config") pod "d7f2ad85-4d53-4814-ad2a-cb8323733682" (UID: "d7f2ad85-4d53-4814-ad2a-cb8323733682"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:51:42.637625 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.637607    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-config\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:51:42.637768 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.637744    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d7f2ad85-4d53-4814-ad2a-cb8323733682" (UID: "d7f2ad85-4d53-4814-ad2a-cb8323733682"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:51:42.637827 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.637740    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-service-ca" (OuterVolumeSpecName: "service-ca") pod "d7f2ad85-4d53-4814-ad2a-cb8323733682" (UID: "d7f2ad85-4d53-4814-ad2a-cb8323733682"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:51:42.639658 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.639633    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d7f2ad85-4d53-4814-ad2a-cb8323733682" (UID: "d7f2ad85-4d53-4814-ad2a-cb8323733682"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:51:42.639753 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.639690    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d7f2ad85-4d53-4814-ad2a-cb8323733682" (UID: "d7f2ad85-4d53-4814-ad2a-cb8323733682"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:51:42.639753 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.639697    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f2ad85-4d53-4814-ad2a-cb8323733682-kube-api-access-hpmct" (OuterVolumeSpecName: "kube-api-access-hpmct") pod "d7f2ad85-4d53-4814-ad2a-cb8323733682" (UID: "d7f2ad85-4d53-4814-ad2a-cb8323733682"). InnerVolumeSpecName "kube-api-access-hpmct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:51:42.738376 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.738313    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpmct\" (UniqueName: \"kubernetes.io/projected/d7f2ad85-4d53-4814-ad2a-cb8323733682-kube-api-access-hpmct\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:51:42.738376 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.738346    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:51:42.738376 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.738356    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7f2ad85-4d53-4814-ad2a-cb8323733682-oauth-serving-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:51:42.738376 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.738364    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-oauth-config\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:51:42.738376 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:42.738372    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7f2ad85-4d53-4814-ad2a-cb8323733682-console-serving-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:51:43.354961 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:43.354928    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:43.355344 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:43.354992    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:51:43.373793 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:43.373770    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64645bfccf-ttnbj"
Sep 11 03:51:43.373933 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:43.373793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64645bfccf-ttnbj" event={"ID":"d7f2ad85-4d53-4814-ad2a-cb8323733682","Type":"ContainerDied","Data":"1a7f3b11a729fa5627e8904eaf3d4803fab9901af42c34b0bc1624cadb321a1c"}
Sep 11 03:51:43.373933 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:43.373850    2581 scope.go:117] "RemoveContainer" containerID="79c796a35705ef517bca3f390c6ebd48f688dfa75a725ef40b175a08911b67da"
Sep 11 03:51:43.393984 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:43.393961    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64645bfccf-ttnbj"]
Sep 11 03:51:43.400368 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:43.399592    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64645bfccf-ttnbj"]
Sep 11 03:51:43.860608 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:43.860573    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7f2ad85-4d53-4814-ad2a-cb8323733682" path="/var/lib/kubelet/pods/d7f2ad85-4d53-4814-ad2a-cb8323733682/volumes"
Sep 11 03:51:44.895128 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:44.895094    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:44.948108 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:44.948083    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:51.186713 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:51.186677    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t7sgk"
Sep 11 03:51:53.444588 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.444554    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f668854cb-zg2gn" podUID="d582a9c9-63bd-48cd-a884-d83c49989227" containerName="console" containerID="cri-o://aa9137539fd252b464699014ddaa312b703fb993918345b58c0060fd032f5d04" gracePeriod=15
Sep 11 03:51:53.728128 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.728102    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:53.829805 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.829769    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-trusted-ca-bundle\") pod \"d582a9c9-63bd-48cd-a884-d83c49989227\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") "
Sep 11 03:51:53.829971 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.829822    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-console-config\") pod \"d582a9c9-63bd-48cd-a884-d83c49989227\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") "
Sep 11 03:51:53.829971 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.829862    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d582a9c9-63bd-48cd-a884-d83c49989227-console-oauth-config\") pod \"d582a9c9-63bd-48cd-a884-d83c49989227\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") "
Sep 11 03:51:53.829971 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.829883    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d582a9c9-63bd-48cd-a884-d83c49989227-console-serving-cert\") pod \"d582a9c9-63bd-48cd-a884-d83c49989227\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") "
Sep 11 03:51:53.829971 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.829910    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn2xt\" (UniqueName: \"kubernetes.io/projected/d582a9c9-63bd-48cd-a884-d83c49989227-kube-api-access-zn2xt\") pod \"d582a9c9-63bd-48cd-a884-d83c49989227\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") "
Sep 11 03:51:53.829971 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.829935    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-oauth-serving-cert\") pod \"d582a9c9-63bd-48cd-a884-d83c49989227\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") "
Sep 11 03:51:53.830257 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.830054    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-service-ca\") pod \"d582a9c9-63bd-48cd-a884-d83c49989227\" (UID: \"d582a9c9-63bd-48cd-a884-d83c49989227\") "
Sep 11 03:51:53.830315 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.830251    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d582a9c9-63bd-48cd-a884-d83c49989227" (UID: "d582a9c9-63bd-48cd-a884-d83c49989227"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:51:53.830398 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.830365    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-console-config" (OuterVolumeSpecName: "console-config") pod "d582a9c9-63bd-48cd-a884-d83c49989227" (UID: "d582a9c9-63bd-48cd-a884-d83c49989227"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:51:53.830450 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.830409    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-trusted-ca-bundle\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:51:53.830450 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.830426    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d582a9c9-63bd-48cd-a884-d83c49989227" (UID: "d582a9c9-63bd-48cd-a884-d83c49989227"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:51:53.830916 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.830884    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-service-ca" (OuterVolumeSpecName: "service-ca") pod "d582a9c9-63bd-48cd-a884-d83c49989227" (UID: "d582a9c9-63bd-48cd-a884-d83c49989227"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:51:53.832192 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.832161    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d582a9c9-63bd-48cd-a884-d83c49989227-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d582a9c9-63bd-48cd-a884-d83c49989227" (UID: "d582a9c9-63bd-48cd-a884-d83c49989227"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:51:53.832298 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.832276    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d582a9c9-63bd-48cd-a884-d83c49989227-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d582a9c9-63bd-48cd-a884-d83c49989227" (UID: "d582a9c9-63bd-48cd-a884-d83c49989227"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:51:53.832339 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.832302    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d582a9c9-63bd-48cd-a884-d83c49989227-kube-api-access-zn2xt" (OuterVolumeSpecName: "kube-api-access-zn2xt") pod "d582a9c9-63bd-48cd-a884-d83c49989227" (UID: "d582a9c9-63bd-48cd-a884-d83c49989227"). InnerVolumeSpecName "kube-api-access-zn2xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:51:53.931149 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.931126    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d582a9c9-63bd-48cd-a884-d83c49989227-console-serving-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:51:53.931245 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.931164    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn2xt\" (UniqueName: \"kubernetes.io/projected/d582a9c9-63bd-48cd-a884-d83c49989227-kube-api-access-zn2xt\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:51:53.931245 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.931173    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-oauth-serving-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:51:53.931245 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.931182    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:51:53.931245 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.931191    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d582a9c9-63bd-48cd-a884-d83c49989227-console-config\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:51:53.931245 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:53.931199    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d582a9c9-63bd-48cd-a884-d83c49989227-console-oauth-config\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:51:54.422891 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:54.422840    2581 generic.go:356] "Generic (PLEG): container finished" podID="d582a9c9-63bd-48cd-a884-d83c49989227" containerID="aa9137539fd252b464699014ddaa312b703fb993918345b58c0060fd032f5d04" exitCode=0
Sep 11 03:51:54.423083 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:54.422900    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f668854cb-zg2gn"
Sep 11 03:51:54.423083 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:54.422902    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f668854cb-zg2gn" event={"ID":"d582a9c9-63bd-48cd-a884-d83c49989227","Type":"ContainerDied","Data":"aa9137539fd252b464699014ddaa312b703fb993918345b58c0060fd032f5d04"}
Sep 11 03:51:54.423083 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:54.423012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f668854cb-zg2gn" event={"ID":"d582a9c9-63bd-48cd-a884-d83c49989227","Type":"ContainerDied","Data":"b73c8594ece8897bfff80f62bd0f7cecdbbf38311f82241970d4722cab49245d"}
Sep 11 03:51:54.423083 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:54.423050    2581 scope.go:117] "RemoveContainer" containerID="aa9137539fd252b464699014ddaa312b703fb993918345b58c0060fd032f5d04"
Sep 11 03:51:54.431243 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:54.431227    2581 scope.go:117] "RemoveContainer" containerID="aa9137539fd252b464699014ddaa312b703fb993918345b58c0060fd032f5d04"
Sep 11 03:51:54.431548 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:51:54.431520    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9137539fd252b464699014ddaa312b703fb993918345b58c0060fd032f5d04\": container with ID starting with aa9137539fd252b464699014ddaa312b703fb993918345b58c0060fd032f5d04 not found: ID does not exist" containerID="aa9137539fd252b464699014ddaa312b703fb993918345b58c0060fd032f5d04"
Sep 11 03:51:54.431597 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:54.431555    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9137539fd252b464699014ddaa312b703fb993918345b58c0060fd032f5d04"} err="failed to get container status \"aa9137539fd252b464699014ddaa312b703fb993918345b58c0060fd032f5d04\": rpc error: code = NotFound desc = could not find container \"aa9137539fd252b464699014ddaa312b703fb993918345b58c0060fd032f5d04\": container with ID starting with aa9137539fd252b464699014ddaa312b703fb993918345b58c0060fd032f5d04 not found: ID does not exist"
Sep 11 03:51:54.438917 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:54.438891    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f668854cb-zg2gn"]
Sep 11 03:51:54.444281 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:54.444262    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f668854cb-zg2gn"]
Sep 11 03:51:55.860369 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:55.860335    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d582a9c9-63bd-48cd-a884-d83c49989227" path="/var/lib/kubelet/pods/d582a9c9-63bd-48cd-a884-d83c49989227/volumes"
Sep 11 03:51:59.893750 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:59.893707    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:51:59.949329 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:51:59.949305    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:00.436229 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:00.436195    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:00.497535 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:00.497503    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:03.360288 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:03.360262    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:52:03.363958 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:03.363935    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-bb4598894-mcwhb"
Sep 11 03:52:04.352945 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.352897    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d655ff58d-g7k9w" podUID="47e759dc-b9ba-4222-8c94-4fab992a6dc8" containerName="console" containerID="cri-o://0438c9688687b97043d52a9a4ad7730abe438b1dfb29e6d3cb400fd116169f11" gracePeriod=15
Sep 11 03:52:04.595138 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.595112    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:52:04.722525 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.722437    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-oauth-config\") pod \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") "
Sep 11 03:52:04.722525 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.722478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-trusted-ca-bundle\") pod \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") "
Sep 11 03:52:04.722525 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.722507    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-serving-cert\") pod \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") "
Sep 11 03:52:04.722790 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.722532    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-config\") pod \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") "
Sep 11 03:52:04.722790 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.722579    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnbws\" (UniqueName: \"kubernetes.io/projected/47e759dc-b9ba-4222-8c94-4fab992a6dc8-kube-api-access-dnbws\") pod \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") "
Sep 11 03:52:04.722790 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.722595    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-service-ca\") pod \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") "
Sep 11 03:52:04.722790 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.722701    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-oauth-serving-cert\") pod \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\" (UID: \"47e759dc-b9ba-4222-8c94-4fab992a6dc8\") "
Sep 11 03:52:04.723036 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.722993    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-config" (OuterVolumeSpecName: "console-config") pod "47e759dc-b9ba-4222-8c94-4fab992a6dc8" (UID: "47e759dc-b9ba-4222-8c94-4fab992a6dc8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:52:04.723099 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.722999    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-service-ca" (OuterVolumeSpecName: "service-ca") pod "47e759dc-b9ba-4222-8c94-4fab992a6dc8" (UID: "47e759dc-b9ba-4222-8c94-4fab992a6dc8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:52:04.723099 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.723053    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "47e759dc-b9ba-4222-8c94-4fab992a6dc8" (UID: "47e759dc-b9ba-4222-8c94-4fab992a6dc8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:52:04.723333 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.723309    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "47e759dc-b9ba-4222-8c94-4fab992a6dc8" (UID: "47e759dc-b9ba-4222-8c94-4fab992a6dc8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:52:04.724747 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.724724    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "47e759dc-b9ba-4222-8c94-4fab992a6dc8" (UID: "47e759dc-b9ba-4222-8c94-4fab992a6dc8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:52:04.725128 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.725109    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e759dc-b9ba-4222-8c94-4fab992a6dc8-kube-api-access-dnbws" (OuterVolumeSpecName: "kube-api-access-dnbws") pod "47e759dc-b9ba-4222-8c94-4fab992a6dc8" (UID: "47e759dc-b9ba-4222-8c94-4fab992a6dc8"). InnerVolumeSpecName "kube-api-access-dnbws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:52:04.725128 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.725117    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "47e759dc-b9ba-4222-8c94-4fab992a6dc8" (UID: "47e759dc-b9ba-4222-8c94-4fab992a6dc8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:52:04.823649 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.823602    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-oauth-config\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:52:04.823649 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.823644    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-trusted-ca-bundle\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:52:04.823649 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.823653    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-serving-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:52:04.823649 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.823663    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-console-config\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:52:04.823899 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.823673    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnbws\" (UniqueName: \"kubernetes.io/projected/47e759dc-b9ba-4222-8c94-4fab992a6dc8-kube-api-access-dnbws\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:52:04.823899 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.823682    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:52:04.823899 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:04.823690    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47e759dc-b9ba-4222-8c94-4fab992a6dc8-oauth-serving-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:52:05.470794 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:05.470751    2581 generic.go:356] "Generic (PLEG): container finished" podID="47e759dc-b9ba-4222-8c94-4fab992a6dc8" containerID="0438c9688687b97043d52a9a4ad7730abe438b1dfb29e6d3cb400fd116169f11" exitCode=0
Sep 11 03:52:05.471115 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:05.470806    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d655ff58d-g7k9w"
Sep 11 03:52:05.471115 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:05.470799    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d655ff58d-g7k9w" event={"ID":"47e759dc-b9ba-4222-8c94-4fab992a6dc8","Type":"ContainerDied","Data":"0438c9688687b97043d52a9a4ad7730abe438b1dfb29e6d3cb400fd116169f11"}
Sep 11 03:52:05.471115 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:05.470860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d655ff58d-g7k9w" event={"ID":"47e759dc-b9ba-4222-8c94-4fab992a6dc8","Type":"ContainerDied","Data":"d0e044bfc599224dea34fe9a87b2c94f67b6439f0278c6b62fa34085a1fe40e7"}
Sep 11 03:52:05.471115 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:05.470879    2581 scope.go:117] "RemoveContainer" containerID="0438c9688687b97043d52a9a4ad7730abe438b1dfb29e6d3cb400fd116169f11"
Sep 11 03:52:05.479373 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:05.479358    2581 scope.go:117] "RemoveContainer" containerID="0438c9688687b97043d52a9a4ad7730abe438b1dfb29e6d3cb400fd116169f11"
Sep 11 03:52:05.479640 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:52:05.479620    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0438c9688687b97043d52a9a4ad7730abe438b1dfb29e6d3cb400fd116169f11\": container with ID starting with 0438c9688687b97043d52a9a4ad7730abe438b1dfb29e6d3cb400fd116169f11 not found: ID does not exist" containerID="0438c9688687b97043d52a9a4ad7730abe438b1dfb29e6d3cb400fd116169f11"
Sep 11 03:52:05.479708 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:05.479652    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0438c9688687b97043d52a9a4ad7730abe438b1dfb29e6d3cb400fd116169f11"} err="failed to get container status \"0438c9688687b97043d52a9a4ad7730abe438b1dfb29e6d3cb400fd116169f11\": rpc error: code = NotFound desc = could not find container \"0438c9688687b97043d52a9a4ad7730abe438b1dfb29e6d3cb400fd116169f11\": container with ID starting with 0438c9688687b97043d52a9a4ad7730abe438b1dfb29e6d3cb400fd116169f11 not found: ID does not exist"
Sep 11 03:52:05.492064 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:05.492044    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d655ff58d-g7k9w"]
Sep 11 03:52:05.495306 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:05.495287    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d655ff58d-g7k9w"]
Sep 11 03:52:05.860667 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:05.860636    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47e759dc-b9ba-4222-8c94-4fab992a6dc8" path="/var/lib/kubelet/pods/47e759dc-b9ba-4222-8c94-4fab992a6dc8/volumes"
Sep 11 03:52:08.379698 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:08.379661    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:08.479731 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:08.479694    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:08.529880 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:08.529852    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:25.167637 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:25.167603    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:52:25.186754 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:25.186731    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:52:25.566077 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:25.565991    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:52:29.351129 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:29.351093    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:34.657152 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:34.657117    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:34.715821 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:34.715792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:36.991271 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:36.991215    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:40.987759 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:40.987724    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:46.579635 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:46.579594    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:46.639105 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:46.639074    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:48.231379 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:48.231341    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:48.282608 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:48.282572    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:58.739564 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:58.739527    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:58.836541 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:58.836508    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:58.893880 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:58.893852    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:52:59.906080 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:52:59.906043    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:53:00.007245 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:53:00.007209    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:53:15.869374 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:53:15.869325    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:53:15.970154 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:53:15.970119    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:04.352764 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:04.352731    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:04.453711 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:04.453681    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:31.312802 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:31.312731    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:32.331748 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:32.331714    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:34.941876 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:34.941840    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:35.042699 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:35.042669    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:36.387912 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:36.387879    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:36.496329 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:36.496299    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:36.608149 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:36.608113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:38.750412 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:38.750371    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:38.948167 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:38.948132    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:39.915130 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.915098    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-czkq9"]
Sep 11 03:54:39.915472 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.915401    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7f2ad85-4d53-4814-ad2a-cb8323733682" containerName="console"
Sep 11 03:54:39.915472 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.915412    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7f2ad85-4d53-4814-ad2a-cb8323733682" containerName="console"
Sep 11 03:54:39.915472 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.915422    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d582a9c9-63bd-48cd-a884-d83c49989227" containerName="console"
Sep 11 03:54:39.915472 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.915428    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d582a9c9-63bd-48cd-a884-d83c49989227" containerName="console"
Sep 11 03:54:39.915472 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.915446    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e759dc-b9ba-4222-8c94-4fab992a6dc8" containerName="console"
Sep 11 03:54:39.915472 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.915451    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e759dc-b9ba-4222-8c94-4fab992a6dc8" containerName="console"
Sep 11 03:54:39.915692 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.915499    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7f2ad85-4d53-4814-ad2a-cb8323733682" containerName="console"
Sep 11 03:54:39.915692 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.915509    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d582a9c9-63bd-48cd-a884-d83c49989227" containerName="console"
Sep 11 03:54:39.915692 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.915517    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="47e759dc-b9ba-4222-8c94-4fab992a6dc8" containerName="console"
Sep 11 03:54:39.918394 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.918376    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-czkq9"
Sep 11 03:54:39.921191 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.921171    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qwcj9\""
Sep 11 03:54:39.921308 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.921212    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 03:54:39.921308 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.921214    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 03:54:39.928390 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:39.928372    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-czkq9"]
Sep 11 03:54:40.004639 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:40.004613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcww2\" (UniqueName: \"kubernetes.io/projected/01ed0809-c057-4d09-8af5-5bc75696364a-kube-api-access-qcww2\") pod \"cert-manager-cainjector-559cdd8c87-czkq9\" (UID: \"01ed0809-c057-4d09-8af5-5bc75696364a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-czkq9"
Sep 11 03:54:40.004764 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:40.004655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01ed0809-c057-4d09-8af5-5bc75696364a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-czkq9\" (UID: \"01ed0809-c057-4d09-8af5-5bc75696364a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-czkq9"
Sep 11 03:54:40.105324 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:40.105298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcww2\" (UniqueName: \"kubernetes.io/projected/01ed0809-c057-4d09-8af5-5bc75696364a-kube-api-access-qcww2\") pod \"cert-manager-cainjector-559cdd8c87-czkq9\" (UID: \"01ed0809-c057-4d09-8af5-5bc75696364a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-czkq9"
Sep 11 03:54:40.105445 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:40.105340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01ed0809-c057-4d09-8af5-5bc75696364a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-czkq9\" (UID: \"01ed0809-c057-4d09-8af5-5bc75696364a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-czkq9"
Sep 11 03:54:40.115928 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:40.115901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01ed0809-c057-4d09-8af5-5bc75696364a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-czkq9\" (UID: \"01ed0809-c057-4d09-8af5-5bc75696364a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-czkq9"
Sep 11 03:54:40.116052 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:40.116012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcww2\" (UniqueName: \"kubernetes.io/projected/01ed0809-c057-4d09-8af5-5bc75696364a-kube-api-access-qcww2\") pod \"cert-manager-cainjector-559cdd8c87-czkq9\" (UID: \"01ed0809-c057-4d09-8af5-5bc75696364a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-czkq9"
Sep 11 03:54:40.236242 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:40.236162    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-czkq9"
Sep 11 03:54:40.508107 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:40.508031    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-czkq9"]
Sep 11 03:54:40.510614 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:54:40.510587    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01ed0809_c057_4d09_8af5_5bc75696364a.slice/crio-54e8583e9db4a147bb711f4812295a6ef2160d1fb13c0beca664ed6b55a7f1d9 WatchSource:0}: Error finding container 54e8583e9db4a147bb711f4812295a6ef2160d1fb13c0beca664ed6b55a7f1d9: Status 404 returned error can't find the container with id 54e8583e9db4a147bb711f4812295a6ef2160d1fb13c0beca664ed6b55a7f1d9
Sep 11 03:54:41.080139 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:41.080104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-czkq9" event={"ID":"01ed0809-c057-4d09-8af5-5bc75696364a","Type":"ContainerStarted","Data":"54e8583e9db4a147bb711f4812295a6ef2160d1fb13c0beca664ed6b55a7f1d9"}
Sep 11 03:54:41.718747 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:41.718707    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:41.782243 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:41.782103    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:44.097290 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:44.097258    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-czkq9" event={"ID":"01ed0809-c057-4d09-8af5-5bc75696364a","Type":"ContainerStarted","Data":"c3cef25583ac8477dda977317016beffa126fe9512f66d54a7ac55b8e89dacf6"}
Sep 11 03:54:44.120298 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:44.120255    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-czkq9" podStartSLOduration=1.8634543300000002 podStartE2EDuration="5.120240693s" totalImagesPullingTime="3.256786363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:44.117634132 +0000 UTC m=+276.740027160" watchObservedRunningTime="2026-09-11 03:54:44.120240693 +0000 UTC m=+276.742633720"
Sep 11 03:54:44.391168 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:44.391095    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sc5w6"]
Sep 11 03:54:44.394543 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:44.394518    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sc5w6"
Sep 11 03:54:44.397872 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:44.397852    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-75mqg\""
Sep 11 03:54:44.409907 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:44.409884    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sc5w6"]
Sep 11 03:54:44.441342 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:44.441316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9vvk\" (UniqueName: \"kubernetes.io/projected/ca83cdeb-bd15-4a21-8988-b492f722ebf9-kube-api-access-p9vvk\") pod \"cert-manager-webhook-6486ffd997-sc5w6\" (UID: \"ca83cdeb-bd15-4a21-8988-b492f722ebf9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sc5w6"
Sep 11 03:54:44.441447 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:44.441374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca83cdeb-bd15-4a21-8988-b492f722ebf9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sc5w6\" (UID: \"ca83cdeb-bd15-4a21-8988-b492f722ebf9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sc5w6"
Sep 11 03:54:44.542341 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:44.542306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9vvk\" (UniqueName: \"kubernetes.io/projected/ca83cdeb-bd15-4a21-8988-b492f722ebf9-kube-api-access-p9vvk\") pod \"cert-manager-webhook-6486ffd997-sc5w6\" (UID: \"ca83cdeb-bd15-4a21-8988-b492f722ebf9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sc5w6"
Sep 11 03:54:44.542532 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:44.542374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca83cdeb-bd15-4a21-8988-b492f722ebf9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sc5w6\" (UID: \"ca83cdeb-bd15-4a21-8988-b492f722ebf9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sc5w6"
Sep 11 03:54:44.551761 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:44.551729    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9vvk\" (UniqueName: \"kubernetes.io/projected/ca83cdeb-bd15-4a21-8988-b492f722ebf9-kube-api-access-p9vvk\") pod \"cert-manager-webhook-6486ffd997-sc5w6\" (UID: \"ca83cdeb-bd15-4a21-8988-b492f722ebf9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sc5w6"
Sep 11 03:54:44.551873 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:44.551731    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca83cdeb-bd15-4a21-8988-b492f722ebf9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sc5w6\" (UID: \"ca83cdeb-bd15-4a21-8988-b492f722ebf9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sc5w6"
Sep 11 03:54:44.704126 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:44.704037    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sc5w6"
Sep 11 03:54:44.974776 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:44.974755    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sc5w6"]
Sep 11 03:54:44.976728 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:54:44.976705    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca83cdeb_bd15_4a21_8988_b492f722ebf9.slice/crio-057f1f19709fb6c03d0ec4e5f142231f2fd3dcf60e5cadc7a4a2466a610b91f5 WatchSource:0}: Error finding container 057f1f19709fb6c03d0ec4e5f142231f2fd3dcf60e5cadc7a4a2466a610b91f5: Status 404 returned error can't find the container with id 057f1f19709fb6c03d0ec4e5f142231f2fd3dcf60e5cadc7a4a2466a610b91f5
Sep 11 03:54:45.102810 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:45.102776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sc5w6" event={"ID":"ca83cdeb-bd15-4a21-8988-b492f722ebf9","Type":"ContainerStarted","Data":"d9229a3570c4bc53ae867cc506e78545735ab67a35836b35cef6645923ae76bb"}
Sep 11 03:54:45.103213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:45.102823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sc5w6" event={"ID":"ca83cdeb-bd15-4a21-8988-b492f722ebf9","Type":"ContainerStarted","Data":"057f1f19709fb6c03d0ec4e5f142231f2fd3dcf60e5cadc7a4a2466a610b91f5"}
Sep 11 03:54:45.103213 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:45.103015    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sc5w6"
Sep 11 03:54:45.128509 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:45.128465    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-sc5w6" podStartSLOduration=1.128451126 podStartE2EDuration="1.128451126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:45.127141844 +0000 UTC m=+277.749534894" watchObservedRunningTime="2026-09-11 03:54:45.128451126 +0000 UTC m=+277.750844153"
Sep 11 03:54:45.242953 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:45.242879    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:45.345615 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:45.345571    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:45.964252 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:45.964220    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:49.924372 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:49.924340    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:50.023337 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:50.023301    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:51.109286 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:51.109259    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sc5w6"
Sep 11 03:54:55.521163 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:55.521125    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:55.623184 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:55.623153    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:54:57.510849 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:54:57.510819    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:07.796182 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:07.796153    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 03:55:09.874272 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:09.874208    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:09.979447 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:09.979409    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:10.511765 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:10.511732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:11.140460 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.140423    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:11.243641 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.243607    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:11.317096 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.317069    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"]
Sep 11 03:55:11.319353 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.319339    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.322485 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.322464    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 03:55:11.322835 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.322817    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cm9z7\""
Sep 11 03:55:11.322923 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.322850    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 03:55:11.325674 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.325654    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 03:55:11.325766 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.325687    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 03:55:11.338915 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.338890    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"]
Sep 11 03:55:11.453680 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.453592    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:11.469637 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.469613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14dd9380-f197-4f2f-8918-5c0147c7eee5-tmp\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.469712 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.469657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14dd9380-f197-4f2f-8918-5c0147c7eee5-manifests\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.469712 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.469680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14dd9380-f197-4f2f-8918-5c0147c7eee5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.469712 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.469705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whgq8\" (UniqueName: \"kubernetes.io/projected/14dd9380-f197-4f2f-8918-5c0147c7eee5-kube-api-access-whgq8\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.469807 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.469769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14dd9380-f197-4f2f-8918-5c0147c7eee5-webhook-cert\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.570352 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.570325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14dd9380-f197-4f2f-8918-5c0147c7eee5-webhook-cert\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.570505 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.570359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14dd9380-f197-4f2f-8918-5c0147c7eee5-tmp\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.570505 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.570405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14dd9380-f197-4f2f-8918-5c0147c7eee5-manifests\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.570505 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.570435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14dd9380-f197-4f2f-8918-5c0147c7eee5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.570505 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.570463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whgq8\" (UniqueName: \"kubernetes.io/projected/14dd9380-f197-4f2f-8918-5c0147c7eee5-kube-api-access-whgq8\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.570833 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.570806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14dd9380-f197-4f2f-8918-5c0147c7eee5-manifests\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.572800 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.572768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14dd9380-f197-4f2f-8918-5c0147c7eee5-tmp\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.572906 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.572823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14dd9380-f197-4f2f-8918-5c0147c7eee5-webhook-cert\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.572906 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.572869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14dd9380-f197-4f2f-8918-5c0147c7eee5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.581833 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.581803    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:11.583003 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.582986    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whgq8\" (UniqueName: \"kubernetes.io/projected/14dd9380-f197-4f2f-8918-5c0147c7eee5-kube-api-access-whgq8\") pod \"opendatahub-operator-controller-manager-97cf4878d-b54js\" (UID: \"14dd9380-f197-4f2f-8918-5c0147c7eee5\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.630242 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.630210    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:11.676726 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.676687    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:11.779598 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.779507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:11.875248 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.875210    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:11.915320 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.915290    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"]
Sep 11 03:55:11.916549 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:55:11.916519    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14dd9380_f197_4f2f_8918_5c0147c7eee5.slice/crio-980f52f892345164c16518ae8dfe685151982a06228ba9195b3ffbcbe35f6022 WatchSource:0}: Error finding container 980f52f892345164c16518ae8dfe685151982a06228ba9195b3ffbcbe35f6022: Status 404 returned error can't find the container with id 980f52f892345164c16518ae8dfe685151982a06228ba9195b3ffbcbe35f6022
Sep 11 03:55:11.918412 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:11.918396    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 03:55:12.211924 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:12.211878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js" event={"ID":"14dd9380-f197-4f2f-8918-5c0147c7eee5","Type":"ContainerStarted","Data":"980f52f892345164c16518ae8dfe685151982a06228ba9195b3ffbcbe35f6022"}
Sep 11 03:55:15.229754 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:15.229720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js" event={"ID":"14dd9380-f197-4f2f-8918-5c0147c7eee5","Type":"ContainerStarted","Data":"2b7c22de75560dc4b6961942d73f51c2ecbf942ca22ceb6da1e4828be3d6f737"}
Sep 11 03:55:16.235499 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:16.235466    2581 generic.go:356] "Generic (PLEG): container finished" podID="14dd9380-f197-4f2f-8918-5c0147c7eee5" containerID="2b7c22de75560dc4b6961942d73f51c2ecbf942ca22ceb6da1e4828be3d6f737" exitCode=0
Sep 11 03:55:16.235874 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:16.235506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js" event={"ID":"14dd9380-f197-4f2f-8918-5c0147c7eee5","Type":"ContainerDied","Data":"2b7c22de75560dc4b6961942d73f51c2ecbf942ca22ceb6da1e4828be3d6f737"}
Sep 11 03:55:17.242087 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:17.242044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js" event={"ID":"14dd9380-f197-4f2f-8918-5c0147c7eee5","Type":"ContainerStarted","Data":"3d5552da59c5294fa74fa766369791bbd7d27fae4c2a8a9bf84d4d7f5fdf9e07"}
Sep 11 03:55:17.242512 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:17.242168    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:17.267844 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:17.267791    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js" podStartSLOduration=2.2583554279999998 podStartE2EDuration="6.267775767s" totalImagesPullingTime="2.945354916s" totalInitContainerRuntime="1.064065423s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:55:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:17.266134125 +0000 UTC m=+309.888527169" watchObservedRunningTime="2026-09-11 03:55:17.267775767 +0000 UTC m=+309.890168793"
Sep 11 03:55:21.380548 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:21.380516    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:21.477208 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:21.477176    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:27.409726 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:27.409684    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:27.537768 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:27.537729    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:28.249168 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:28.249140    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-b54js"
Sep 11 03:55:29.609795 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.609758    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"]
Sep 11 03:55:29.614966 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.614946    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:29.617575 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.617555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fbdbefa-89ef-46b7-ba15-e566c16a5071-cert\") pod \"lws-controller-manager-67f66d78bd-4fmbs\" (UID: \"4fbdbefa-89ef-46b7-ba15-e566c16a5071\") " pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:29.617696 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.617586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4fbdbefa-89ef-46b7-ba15-e566c16a5071-manager-config\") pod \"lws-controller-manager-67f66d78bd-4fmbs\" (UID: \"4fbdbefa-89ef-46b7-ba15-e566c16a5071\") " pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:29.617696 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.617606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fbdbefa-89ef-46b7-ba15-e566c16a5071-metrics-cert\") pod \"lws-controller-manager-67f66d78bd-4fmbs\" (UID: \"4fbdbefa-89ef-46b7-ba15-e566c16a5071\") " pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:29.617803 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.617742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv87v\" (UniqueName: \"kubernetes.io/projected/4fbdbefa-89ef-46b7-ba15-e566c16a5071-kube-api-access-rv87v\") pod \"lws-controller-manager-67f66d78bd-4fmbs\" (UID: \"4fbdbefa-89ef-46b7-ba15-e566c16a5071\") " pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:29.617874 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.617854    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 03:55:29.617912 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.617861    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 03:55:29.619069 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.619054    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:55:29.619120 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.619059    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 03:55:29.621266 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.621245    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5d9k8\""
Sep 11 03:55:29.621371 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.621284    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 03:55:29.635226 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.635205    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"]
Sep 11 03:55:29.718524 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.718496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fbdbefa-89ef-46b7-ba15-e566c16a5071-cert\") pod \"lws-controller-manager-67f66d78bd-4fmbs\" (UID: \"4fbdbefa-89ef-46b7-ba15-e566c16a5071\") " pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:29.718644 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.718533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4fbdbefa-89ef-46b7-ba15-e566c16a5071-manager-config\") pod \"lws-controller-manager-67f66d78bd-4fmbs\" (UID: \"4fbdbefa-89ef-46b7-ba15-e566c16a5071\") " pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:29.718644 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.718552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fbdbefa-89ef-46b7-ba15-e566c16a5071-metrics-cert\") pod \"lws-controller-manager-67f66d78bd-4fmbs\" (UID: \"4fbdbefa-89ef-46b7-ba15-e566c16a5071\") " pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:29.718644 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.718604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv87v\" (UniqueName: \"kubernetes.io/projected/4fbdbefa-89ef-46b7-ba15-e566c16a5071-kube-api-access-rv87v\") pod \"lws-controller-manager-67f66d78bd-4fmbs\" (UID: \"4fbdbefa-89ef-46b7-ba15-e566c16a5071\") " pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:29.719262 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.719241    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4fbdbefa-89ef-46b7-ba15-e566c16a5071-manager-config\") pod \"lws-controller-manager-67f66d78bd-4fmbs\" (UID: \"4fbdbefa-89ef-46b7-ba15-e566c16a5071\") " pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:29.721001 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.720982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fbdbefa-89ef-46b7-ba15-e566c16a5071-cert\") pod \"lws-controller-manager-67f66d78bd-4fmbs\" (UID: \"4fbdbefa-89ef-46b7-ba15-e566c16a5071\") " pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:29.721120 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.721073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fbdbefa-89ef-46b7-ba15-e566c16a5071-metrics-cert\") pod \"lws-controller-manager-67f66d78bd-4fmbs\" (UID: \"4fbdbefa-89ef-46b7-ba15-e566c16a5071\") " pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:29.732217 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.732198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv87v\" (UniqueName: \"kubernetes.io/projected/4fbdbefa-89ef-46b7-ba15-e566c16a5071-kube-api-access-rv87v\") pod \"lws-controller-manager-67f66d78bd-4fmbs\" (UID: \"4fbdbefa-89ef-46b7-ba15-e566c16a5071\") " pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:29.901436 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.901357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:29.924386 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:29.924359    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:30.041964 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:30.041936    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:30.205945 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:30.205922    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"]
Sep 11 03:55:30.208249 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:55:30.208223    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbdbefa_89ef_46b7_ba15_e566c16a5071.slice/crio-b03ac6156ea6980a2b7fc08d77e474582940acd90036d552d5f0f113964db966 WatchSource:0}: Error finding container b03ac6156ea6980a2b7fc08d77e474582940acd90036d552d5f0f113964db966: Status 404 returned error can't find the container with id b03ac6156ea6980a2b7fc08d77e474582940acd90036d552d5f0f113964db966
Sep 11 03:55:30.299499 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:30.299467    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs" event={"ID":"4fbdbefa-89ef-46b7-ba15-e566c16a5071","Type":"ContainerStarted","Data":"b03ac6156ea6980a2b7fc08d77e474582940acd90036d552d5f0f113964db966"}
Sep 11 03:55:32.965819 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:32.965779    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:33.000296 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.000267    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"]
Sep 11 03:55:33.003753 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.003733    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"
Sep 11 03:55:33.006562 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.006537    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 03:55:33.006655 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.006566    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 03:55:33.006705 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.006683    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 03:55:33.007666 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.007649    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 03:55:33.007727 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.007651    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-67rtg\""
Sep 11 03:55:33.014358 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.014337    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"]
Sep 11 03:55:33.048867 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.048842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16f0b644-f36e-474d-bb69-78158b5d9dfe-tls-certs\") pod \"kube-auth-proxy-5d5f7bb976-7h66z\" (UID: \"16f0b644-f36e-474d-bb69-78158b5d9dfe\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"
Sep 11 03:55:33.048969 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.048871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16f0b644-f36e-474d-bb69-78158b5d9dfe-tmp\") pod \"kube-auth-proxy-5d5f7bb976-7h66z\" (UID: \"16f0b644-f36e-474d-bb69-78158b5d9dfe\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"
Sep 11 03:55:33.048969 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.048956    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vmrm\" (UniqueName: \"kubernetes.io/projected/16f0b644-f36e-474d-bb69-78158b5d9dfe-kube-api-access-4vmrm\") pod \"kube-auth-proxy-5d5f7bb976-7h66z\" (UID: \"16f0b644-f36e-474d-bb69-78158b5d9dfe\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"
Sep 11 03:55:33.149304 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.149278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16f0b644-f36e-474d-bb69-78158b5d9dfe-tls-certs\") pod \"kube-auth-proxy-5d5f7bb976-7h66z\" (UID: \"16f0b644-f36e-474d-bb69-78158b5d9dfe\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"
Sep 11 03:55:33.149304 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.149303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16f0b644-f36e-474d-bb69-78158b5d9dfe-tmp\") pod \"kube-auth-proxy-5d5f7bb976-7h66z\" (UID: \"16f0b644-f36e-474d-bb69-78158b5d9dfe\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"
Sep 11 03:55:33.149449 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.149354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vmrm\" (UniqueName: \"kubernetes.io/projected/16f0b644-f36e-474d-bb69-78158b5d9dfe-kube-api-access-4vmrm\") pod \"kube-auth-proxy-5d5f7bb976-7h66z\" (UID: \"16f0b644-f36e-474d-bb69-78158b5d9dfe\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"
Sep 11 03:55:33.149449 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:55:33.149413    2581 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 11 03:55:33.149564 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:55:33.149480    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16f0b644-f36e-474d-bb69-78158b5d9dfe-tls-certs podName:16f0b644-f36e-474d-bb69-78158b5d9dfe nodeName:}" failed. No retries permitted until 2026-09-11 03:55:33.64946474 +0000 UTC m=+326.271857760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/16f0b644-f36e-474d-bb69-78158b5d9dfe-tls-certs") pod "kube-auth-proxy-5d5f7bb976-7h66z" (UID: "16f0b644-f36e-474d-bb69-78158b5d9dfe") : secret "kube-auth-proxy-tls" not found
Sep 11 03:55:33.151417 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.151398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16f0b644-f36e-474d-bb69-78158b5d9dfe-tmp\") pod \"kube-auth-proxy-5d5f7bb976-7h66z\" (UID: \"16f0b644-f36e-474d-bb69-78158b5d9dfe\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"
Sep 11 03:55:33.159945 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.159924    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vmrm\" (UniqueName: \"kubernetes.io/projected/16f0b644-f36e-474d-bb69-78158b5d9dfe-kube-api-access-4vmrm\") pod \"kube-auth-proxy-5d5f7bb976-7h66z\" (UID: \"16f0b644-f36e-474d-bb69-78158b5d9dfe\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"
Sep 11 03:55:33.316402 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.316363    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs" event={"ID":"4fbdbefa-89ef-46b7-ba15-e566c16a5071","Type":"ContainerStarted","Data":"c4443e383a6fadb5e96c837e322c1b6a3897816ba10b54566ae70dea1e9359bd"}
Sep 11 03:55:33.316539 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.316421    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:33.354283 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.354232    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:33.360847 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.360810    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs" podStartSLOduration=2.04360623 podStartE2EDuration="4.360796834s" totalImagesPullingTime="2.317190604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:33.358745282 +0000 UTC m=+325.981138320" watchObservedRunningTime="2026-09-11 03:55:33.360796834 +0000 UTC m=+325.983189860"
Sep 11 03:55:33.655259 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.655164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16f0b644-f36e-474d-bb69-78158b5d9dfe-tls-certs\") pod \"kube-auth-proxy-5d5f7bb976-7h66z\" (UID: \"16f0b644-f36e-474d-bb69-78158b5d9dfe\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"
Sep 11 03:55:33.657603 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.657580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16f0b644-f36e-474d-bb69-78158b5d9dfe-tls-certs\") pod \"kube-auth-proxy-5d5f7bb976-7h66z\" (UID: \"16f0b644-f36e-474d-bb69-78158b5d9dfe\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"
Sep 11 03:55:33.914348 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:33.914272    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"
Sep 11 03:55:34.201820 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:34.201795    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z"]
Sep 11 03:55:34.203608 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:55:34.203579    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f0b644_f36e_474d_bb69_78158b5d9dfe.slice/crio-e674efa0de7ad208d2dc5bffab0cecef936b32ebd04447a1e57d2ae5b8e235ec WatchSource:0}: Error finding container e674efa0de7ad208d2dc5bffab0cecef936b32ebd04447a1e57d2ae5b8e235ec: Status 404 returned error can't find the container with id e674efa0de7ad208d2dc5bffab0cecef936b32ebd04447a1e57d2ae5b8e235ec
Sep 11 03:55:34.259901 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:34.259879    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:34.322950 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:34.322922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z" event={"ID":"16f0b644-f36e-474d-bb69-78158b5d9dfe","Type":"ContainerStarted","Data":"e674efa0de7ad208d2dc5bffab0cecef936b32ebd04447a1e57d2ae5b8e235ec"}
Sep 11 03:55:34.345376 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:34.345338    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:35.214813 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:35.214774    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:36.237123 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:36.237076    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:36.331888 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:36.331843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:37.340829 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:37.340795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z" event={"ID":"16f0b644-f36e-474d-bb69-78158b5d9dfe","Type":"ContainerStarted","Data":"b0fda76ae4491054ffb4707ae0ff778a972340b11a3749cd343595093edb8c1c"}
Sep 11 03:55:37.355951 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:37.355901    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-7h66z" podStartSLOduration=2.761314766 podStartE2EDuration="5.355885177s" totalImagesPullingTime="2.594570411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:37.355355181 +0000 UTC m=+329.977748229" watchObservedRunningTime="2026-09-11 03:55:37.355885177 +0000 UTC m=+329.978278210"
Sep 11 03:55:38.180032 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:38.179981    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:38.237114 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:38.237080    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:39.318311 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:39.318278    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:39.423939 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:39.423908    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:40.201551 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:40.201522    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:40.425776 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:40.425742    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:41.096534 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:41.096504    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:41.215545 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:41.215506    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:41.314928 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:41.314894    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:41.478637 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:41.478561    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:41.694013 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:41.693978    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:42.480043 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:42.479990    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:44.325625 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:44.325596    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-67f66d78bd-4fmbs"
Sep 11 03:55:50.780210 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:50.780174    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:50.971432 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:50.971392    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:51.060548 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:51.060524    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:51.400774 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:51.400694    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:52.142639 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:52.142587    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:52.245539 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:52.245507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:55:52.308054 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:55:52.307996    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:56:05.090761 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.090686    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c569b9647-d4pk9"]
Sep 11 03:56:05.098899 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.098875    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.102821 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.102791    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 03:56:05.102821 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.102796    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sq84r\""
Sep 11 03:56:05.103894 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.103876    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 03:56:05.103981 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.103918    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 03:56:05.104071 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.104047    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 03:56:05.104141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.104056    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 03:56:05.104419 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.104395    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 03:56:05.105612 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.105582    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 03:56:05.107999 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.107980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c569b9647-d4pk9"]
Sep 11 03:56:05.110831 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.110811    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 03:56:05.150169 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.150134    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:56:05.206293 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.206257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:56:05.233331 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.233301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdff324d-d735-4f06-9dd9-2a31b4a2e597-console-config\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.233454 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.233366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdff324d-d735-4f06-9dd9-2a31b4a2e597-console-serving-cert\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.233454 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.233417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdff324d-d735-4f06-9dd9-2a31b4a2e597-console-oauth-config\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.233454 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.233444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdff324d-d735-4f06-9dd9-2a31b4a2e597-oauth-serving-cert\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.233581 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.233481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdff324d-d735-4f06-9dd9-2a31b4a2e597-service-ca\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.233622 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.233591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdff324d-d735-4f06-9dd9-2a31b4a2e597-trusted-ca-bundle\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.233672 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.233622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz5kc\" (UniqueName: \"kubernetes.io/projected/fdff324d-d735-4f06-9dd9-2a31b4a2e597-kube-api-access-hz5kc\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.334761 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.334724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdff324d-d735-4f06-9dd9-2a31b4a2e597-trusted-ca-bundle\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.334920 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.334767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz5kc\" (UniqueName: \"kubernetes.io/projected/fdff324d-d735-4f06-9dd9-2a31b4a2e597-kube-api-access-hz5kc\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.334920 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.334801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdff324d-d735-4f06-9dd9-2a31b4a2e597-console-config\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.334920 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.334860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdff324d-d735-4f06-9dd9-2a31b4a2e597-console-serving-cert\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.334920 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.334892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdff324d-d735-4f06-9dd9-2a31b4a2e597-console-oauth-config\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.335156 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.334917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdff324d-d735-4f06-9dd9-2a31b4a2e597-oauth-serving-cert\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.335156 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.334967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdff324d-d735-4f06-9dd9-2a31b4a2e597-service-ca\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.335607 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.335575    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdff324d-d735-4f06-9dd9-2a31b4a2e597-console-config\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.335736 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.335675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdff324d-d735-4f06-9dd9-2a31b4a2e597-oauth-serving-cert\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.335736 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.335693    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdff324d-d735-4f06-9dd9-2a31b4a2e597-service-ca\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.335907 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.335883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdff324d-d735-4f06-9dd9-2a31b4a2e597-trusted-ca-bundle\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.337266 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.337248    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdff324d-d735-4f06-9dd9-2a31b4a2e597-console-oauth-config\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.337407 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.337390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdff324d-d735-4f06-9dd9-2a31b4a2e597-console-serving-cert\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.344618 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.344555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz5kc\" (UniqueName: \"kubernetes.io/projected/fdff324d-d735-4f06-9dd9-2a31b4a2e597-kube-api-access-hz5kc\") pod \"console-6c569b9647-d4pk9\" (UID: \"fdff324d-d735-4f06-9dd9-2a31b4a2e597\") " pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.409566 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.409534    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:05.684066 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:05.684038    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c569b9647-d4pk9"]
Sep 11 03:56:05.685849 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:56:05.685822    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdff324d_d735_4f06_9dd9_2a31b4a2e597.slice/crio-0813cb9d673172fd35cab9887b5002e2d31f33617938c5708543d64d290c21e6 WatchSource:0}: Error finding container 0813cb9d673172fd35cab9887b5002e2d31f33617938c5708543d64d290c21e6: Status 404 returned error can't find the container with id 0813cb9d673172fd35cab9887b5002e2d31f33617938c5708543d64d290c21e6
Sep 11 03:56:06.480611 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:06.480577    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c569b9647-d4pk9" event={"ID":"fdff324d-d735-4f06-9dd9-2a31b4a2e597","Type":"ContainerStarted","Data":"ec3dd2ca18ed22b6af33d9113611e56e164752573dd312da206fbdb72ec30ffc"}
Sep 11 03:56:06.480974 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:06.480625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c569b9647-d4pk9" event={"ID":"fdff324d-d735-4f06-9dd9-2a31b4a2e597","Type":"ContainerStarted","Data":"0813cb9d673172fd35cab9887b5002e2d31f33617938c5708543d64d290c21e6"}
Sep 11 03:56:06.500503 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:06.500457    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c569b9647-d4pk9" podStartSLOduration=1.500440839 podStartE2EDuration="1.500440839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:56:06.499815904 +0000 UTC m=+359.122208995" watchObservedRunningTime="2026-09-11 03:56:06.500440839 +0000 UTC m=+359.122834263"
Sep 11 03:56:15.410382 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:15.410348    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:15.410876 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:15.410609    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:15.415199 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:15.415177    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:15.524427 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:15.524402    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c569b9647-d4pk9"
Sep 11 03:56:15.739131 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:15.739051    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:56:15.822569 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:15.822533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:56:19.577661 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:19.577625    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:56:19.677799 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:19.677762    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:56:41.908219 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:41.908173    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:56:42.004993 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:42.004961    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:56:55.519455 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:55.519413    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:56:56.332954 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:56.332916    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:56:57.138320 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:56:57.138290    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:00.550096 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:00.550058    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:00.637736 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:00.637690    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:02.134538 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:02.134502    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:02.540524 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:02.540439    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:03.587623 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:03.587571    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:05.226766 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:05.226734    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:05.773335 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:05.773301    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:06.824136 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:06.824103    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8lppl"]
Sep 11 03:57:06.827384 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:06.827367    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8lppl"
Sep 11 03:57:06.829846 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:06.829820    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 03:57:06.829846 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:06.829841    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 03:57:06.830044 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:06.829841    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-twqsp\""
Sep 11 03:57:06.830044 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:06.829820    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 03:57:06.836933 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:06.836902    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8lppl"]
Sep 11 03:57:06.869840 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:06.869803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbjdk\" (UniqueName: \"kubernetes.io/projected/3003aee5-dd39-4517-9d87-d8fe2e824cc4-kube-api-access-gbjdk\") pod \"dns-operator-controller-manager-7b84cd7c9c-8lppl\" (UID: \"3003aee5-dd39-4517-9d87-d8fe2e824cc4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8lppl"
Sep 11 03:57:06.971331 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:06.971283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbjdk\" (UniqueName: \"kubernetes.io/projected/3003aee5-dd39-4517-9d87-d8fe2e824cc4-kube-api-access-gbjdk\") pod \"dns-operator-controller-manager-7b84cd7c9c-8lppl\" (UID: \"3003aee5-dd39-4517-9d87-d8fe2e824cc4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8lppl"
Sep 11 03:57:06.979598 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:06.979573    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbjdk\" (UniqueName: \"kubernetes.io/projected/3003aee5-dd39-4517-9d87-d8fe2e824cc4-kube-api-access-gbjdk\") pod \"dns-operator-controller-manager-7b84cd7c9c-8lppl\" (UID: \"3003aee5-dd39-4517-9d87-d8fe2e824cc4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8lppl"
Sep 11 03:57:07.138374 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.138279    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8lppl"
Sep 11 03:57:07.178705 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.178671    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:07.347963 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.347932    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"]
Sep 11 03:57:07.351369 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.351351    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"
Sep 11 03:57:07.354005 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.353981    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 03:57:07.354136 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.354093    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 03:57:07.354204 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.354147    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7dpgb\""
Sep 11 03:57:07.354444 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.354416    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:07.358972 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.358951    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"]
Sep 11 03:57:07.374132 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.374113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4pvt\" (UniqueName: \"kubernetes.io/projected/3a97df84-4421-44de-b006-a31aeefd96a7-kube-api-access-x4pvt\") pod \"kuadrant-console-plugin-fbdd4f445-dx2tc\" (UID: \"3a97df84-4421-44de-b006-a31aeefd96a7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"
Sep 11 03:57:07.374228 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.374214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3a97df84-4421-44de-b006-a31aeefd96a7-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dx2tc\" (UID: \"3a97df84-4421-44de-b006-a31aeefd96a7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"
Sep 11 03:57:07.374269 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.374254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a97df84-4421-44de-b006-a31aeefd96a7-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dx2tc\" (UID: \"3a97df84-4421-44de-b006-a31aeefd96a7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"
Sep 11 03:57:07.408794 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.408773    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8lppl"]
Sep 11 03:57:07.411162 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:57:07.411139    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3003aee5_dd39_4517_9d87_d8fe2e824cc4.slice/crio-ba703fd6b1d6ade7ea1fc086195d58531e84c8d0d42f7785317499f6c846cee1 WatchSource:0}: Error finding container ba703fd6b1d6ade7ea1fc086195d58531e84c8d0d42f7785317499f6c846cee1: Status 404 returned error can't find the container with id ba703fd6b1d6ade7ea1fc086195d58531e84c8d0d42f7785317499f6c846cee1
Sep 11 03:57:07.474791 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.474764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a97df84-4421-44de-b006-a31aeefd96a7-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dx2tc\" (UID: \"3a97df84-4421-44de-b006-a31aeefd96a7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"
Sep 11 03:57:07.474890 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.474805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4pvt\" (UniqueName: \"kubernetes.io/projected/3a97df84-4421-44de-b006-a31aeefd96a7-kube-api-access-x4pvt\") pod \"kuadrant-console-plugin-fbdd4f445-dx2tc\" (UID: \"3a97df84-4421-44de-b006-a31aeefd96a7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"
Sep 11 03:57:07.474890 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.474865    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3a97df84-4421-44de-b006-a31aeefd96a7-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dx2tc\" (UID: \"3a97df84-4421-44de-b006-a31aeefd96a7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"
Sep 11 03:57:07.474963 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:57:07.474904    2581 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 11 03:57:07.474996 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:57:07.474971    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a97df84-4421-44de-b006-a31aeefd96a7-plugin-serving-cert podName:3a97df84-4421-44de-b006-a31aeefd96a7 nodeName:}" failed. No retries permitted until 2026-09-11 03:57:07.974951508 +0000 UTC m=+420.597344527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/3a97df84-4421-44de-b006-a31aeefd96a7-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-dx2tc" (UID: "3a97df84-4421-44de-b006-a31aeefd96a7") : secret "plugin-serving-cert" not found
Sep 11 03:57:07.475442 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.475426    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3a97df84-4421-44de-b006-a31aeefd96a7-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dx2tc\" (UID: \"3a97df84-4421-44de-b006-a31aeefd96a7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"
Sep 11 03:57:07.489667 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.489636    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4pvt\" (UniqueName: \"kubernetes.io/projected/3a97df84-4421-44de-b006-a31aeefd96a7-kube-api-access-x4pvt\") pod \"kuadrant-console-plugin-fbdd4f445-dx2tc\" (UID: \"3a97df84-4421-44de-b006-a31aeefd96a7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"
Sep 11 03:57:07.758802 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.758725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8lppl" event={"ID":"3003aee5-dd39-4517-9d87-d8fe2e824cc4","Type":"ContainerStarted","Data":"ba703fd6b1d6ade7ea1fc086195d58531e84c8d0d42f7785317499f6c846cee1"}
Sep 11 03:57:07.977782 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:07.977742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a97df84-4421-44de-b006-a31aeefd96a7-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dx2tc\" (UID: \"3a97df84-4421-44de-b006-a31aeefd96a7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"
Sep 11 03:57:07.978186 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:57:07.977875    2581 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 11 03:57:07.978186 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:57:07.977936    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a97df84-4421-44de-b006-a31aeefd96a7-plugin-serving-cert podName:3a97df84-4421-44de-b006-a31aeefd96a7 nodeName:}" failed. No retries permitted until 2026-09-11 03:57:08.977920907 +0000 UTC m=+421.600313916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/3a97df84-4421-44de-b006-a31aeefd96a7-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-dx2tc" (UID: "3a97df84-4421-44de-b006-a31aeefd96a7") : secret "plugin-serving-cert" not found
Sep 11 03:57:08.241570 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:08.241536    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:08.986749 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:08.986636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a97df84-4421-44de-b006-a31aeefd96a7-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dx2tc\" (UID: \"3a97df84-4421-44de-b006-a31aeefd96a7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"
Sep 11 03:57:08.989676 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:08.989652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a97df84-4421-44de-b006-a31aeefd96a7-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dx2tc\" (UID: \"3a97df84-4421-44de-b006-a31aeefd96a7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"
Sep 11 03:57:09.165548 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:09.165515    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7dpgb\""
Sep 11 03:57:09.173775 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:09.173734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"
Sep 11 03:57:10.204719 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:10.204692    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 03:57:10.403699 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:10.403591    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc"]
Sep 11 03:57:10.406181 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:57:10.406155    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a97df84_4421_44de_b006_a31aeefd96a7.slice/crio-85089fa01c62e5b0736f7c0288cb66c9f80b636aa189c368487f167ff6d6f776 WatchSource:0}: Error finding container 85089fa01c62e5b0736f7c0288cb66c9f80b636aa189c368487f167ff6d6f776: Status 404 returned error can't find the container with id 85089fa01c62e5b0736f7c0288cb66c9f80b636aa189c368487f167ff6d6f776
Sep 11 03:57:10.533603 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:10.533564    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:10.633640 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:10.633606    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:10.773826 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:10.773747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc" event={"ID":"3a97df84-4421-44de-b006-a31aeefd96a7","Type":"ContainerStarted","Data":"85089fa01c62e5b0736f7c0288cb66c9f80b636aa189c368487f167ff6d6f776"}
Sep 11 03:57:10.775367 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:10.775343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8lppl" event={"ID":"3003aee5-dd39-4517-9d87-d8fe2e824cc4","Type":"ContainerStarted","Data":"abb47b706f15a0e90e9ac51f2e0a0f9dac2dbda88c2621dd99dd6a4b48553e7c"}
Sep 11 03:57:10.775500 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:10.775478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8lppl"
Sep 11 03:57:10.794515 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:10.794469    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8lppl" podStartSLOduration=2.005834573 podStartE2EDuration="4.794455226s" totalImagesPullingTime="2.788620653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:10.793413352 +0000 UTC m=+423.415806411" watchObservedRunningTime="2026-09-11 03:57:10.794455226 +0000 UTC m=+423.416848253"
Sep 11 03:57:11.146173 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:11.146139    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:11.245557 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:11.245519    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:11.540195 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:11.540113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:11.603500 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:11.603392    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:14.031304 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:14.031266    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:15.801959 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:15.801922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc" event={"ID":"3a97df84-4421-44de-b006-a31aeefd96a7","Type":"ContainerStarted","Data":"745cf77a6f0ca51ab1cc073dc219870d2bf8028f9d32eee9677424d3f5498a73"}
Sep 11 03:57:15.818631 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:15.818584    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dx2tc" podStartSLOduration=4.20374179 podStartE2EDuration="8.818570673s" totalImagesPullingTime="4.614828883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:15.817721819 +0000 UTC m=+428.440114847" watchObservedRunningTime="2026-09-11 03:57:15.818570673 +0000 UTC m=+428.440963700"
Sep 11 03:57:18.001241 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:18.001205    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:18.097703 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:18.097678    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:19.775878 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:19.775841    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"]
Sep 11 03:57:19.779910 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:19.779888    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"
Sep 11 03:57:19.783832 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:19.783811    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-762n8\""
Sep 11 03:57:19.799198 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:19.799169    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"]
Sep 11 03:57:19.888960 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:19.888932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hn5\" (UniqueName: \"kubernetes.io/projected/2b9b9624-1cf9-4d38-bd62-0c445e98b087-kube-api-access-69hn5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fnh8f\" (UID: \"2b9b9624-1cf9-4d38-bd62-0c445e98b087\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"
Sep 11 03:57:19.889151 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:19.889062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b9b9624-1cf9-4d38-bd62-0c445e98b087-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fnh8f\" (UID: \"2b9b9624-1cf9-4d38-bd62-0c445e98b087\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"
Sep 11 03:57:19.989984 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:19.989952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hn5\" (UniqueName: \"kubernetes.io/projected/2b9b9624-1cf9-4d38-bd62-0c445e98b087-kube-api-access-69hn5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fnh8f\" (UID: \"2b9b9624-1cf9-4d38-bd62-0c445e98b087\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"
Sep 11 03:57:19.990153 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:19.990013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b9b9624-1cf9-4d38-bd62-0c445e98b087-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fnh8f\" (UID: \"2b9b9624-1cf9-4d38-bd62-0c445e98b087\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"
Sep 11 03:57:19.990375 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:19.990357    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b9b9624-1cf9-4d38-bd62-0c445e98b087-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fnh8f\" (UID: \"2b9b9624-1cf9-4d38-bd62-0c445e98b087\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"
Sep 11 03:57:20.006622 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:20.006594    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hn5\" (UniqueName: \"kubernetes.io/projected/2b9b9624-1cf9-4d38-bd62-0c445e98b087-kube-api-access-69hn5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fnh8f\" (UID: \"2b9b9624-1cf9-4d38-bd62-0c445e98b087\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"
Sep 11 03:57:20.090317 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:20.090292    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"
Sep 11 03:57:20.366637 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:20.366566    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"]
Sep 11 03:57:20.370045 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:57:20.369993    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9b9624_1cf9_4d38_bd62_0c445e98b087.slice/crio-825cfa23e49eaac4ed7d8653d45c08ec94df59ae09738d14f692b8aedbe96c7f WatchSource:0}: Error finding container 825cfa23e49eaac4ed7d8653d45c08ec94df59ae09738d14f692b8aedbe96c7f: Status 404 returned error can't find the container with id 825cfa23e49eaac4ed7d8653d45c08ec94df59ae09738d14f692b8aedbe96c7f
Sep 11 03:57:20.825924 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:20.825892    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f" event={"ID":"2b9b9624-1cf9-4d38-bd62-0c445e98b087","Type":"ContainerStarted","Data":"825cfa23e49eaac4ed7d8653d45c08ec94df59ae09738d14f692b8aedbe96c7f"}
Sep 11 03:57:21.782889 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:21.782861    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8lppl"
Sep 11 03:57:22.769981 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:22.769939    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:22.869441 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:22.869397    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:25.257747 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:25.257715    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:25.336594 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:25.336550    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:25.854500 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:25.854464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f" event={"ID":"2b9b9624-1cf9-4d38-bd62-0c445e98b087","Type":"ContainerStarted","Data":"d9806b12c1b785c5f89376df67234b7fce2c20c19c509651325dc61783b89a8e"}
Sep 11 03:57:25.854690 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:25.854539    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"
Sep 11 03:57:25.880465 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:25.880406    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f" podStartSLOduration=2.088242703 podStartE2EDuration="6.880391753s" totalImagesPullingTime="4.79214905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:25.877132515 +0000 UTC m=+438.499525554" watchObservedRunningTime="2026-09-11 03:57:25.880391753 +0000 UTC m=+438.502784779"
Sep 11 03:57:31.796375 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:31.796278    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:31.893620 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:31.893593    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:31.950587 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:31.950547    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:36.861131 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:36.861101    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"
Sep 11 03:57:38.810154 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:38.810123    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:38.912937 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:38.912903    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:43.614392 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:43.614351    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:43.714866 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:43.714829    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:46.062184 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:46.062149    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:46.163370 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:46.163332    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:51.856617 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:51.856580    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:51.912043 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:51.911997    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:52.814268 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:52.814234    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:52.951071 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:52.951008    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:53.213056 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:53.212946    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:53.313475 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:53.313441    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:53.409479 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:53.409448    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:53.623674 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:53.623641    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:53.691800 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:53.691769    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:53.894724 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:53.894641    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:59.602890 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:59.602854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:59.715539 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:59.715486    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:57:59.787455 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:57:59.787419    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:00.230281 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:00.230243    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:00.335341 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:00.335293    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:00.483807 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:00.483722    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b724s"]
Sep 11 03:58:00.503124 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:00.503095    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b724s"]
Sep 11 03:58:00.503302 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:00.503181    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b724s"
Sep 11 03:58:00.505543 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:00.505521    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-r9bmw\""
Sep 11 03:58:00.519242 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:00.519220    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:00.543224 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:00.543196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zntld\" (UniqueName: \"kubernetes.io/projected/624a6612-1d12-4e50-ba0d-07fdbdbde800-kube-api-access-zntld\") pod \"authorino-7bcfd8f9cd-b724s\" (UID: \"624a6612-1d12-4e50-ba0d-07fdbdbde800\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b724s"
Sep 11 03:58:00.578173 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:00.578144    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:00.644594 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:00.644564    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zntld\" (UniqueName: \"kubernetes.io/projected/624a6612-1d12-4e50-ba0d-07fdbdbde800-kube-api-access-zntld\") pod \"authorino-7bcfd8f9cd-b724s\" (UID: \"624a6612-1d12-4e50-ba0d-07fdbdbde800\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b724s"
Sep 11 03:58:00.658676 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:00.658648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zntld\" (UniqueName: \"kubernetes.io/projected/624a6612-1d12-4e50-ba0d-07fdbdbde800-kube-api-access-zntld\") pod \"authorino-7bcfd8f9cd-b724s\" (UID: \"624a6612-1d12-4e50-ba0d-07fdbdbde800\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b724s"
Sep 11 03:58:00.814820 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:00.814792    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b724s"
Sep 11 03:58:01.080645 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:01.080618    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b724s"]
Sep 11 03:58:01.082647 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:58:01.082611    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod624a6612_1d12_4e50_ba0d_07fdbdbde800.slice/crio-e89d17abd174994dd48c517810b03febd8b7f08b66006598cb3d27717c92abc5 WatchSource:0}: Error finding container e89d17abd174994dd48c517810b03febd8b7f08b66006598cb3d27717c92abc5: Status 404 returned error can't find the container with id e89d17abd174994dd48c517810b03febd8b7f08b66006598cb3d27717c92abc5
Sep 11 03:58:02.024773 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:02.024734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b724s" event={"ID":"624a6612-1d12-4e50-ba0d-07fdbdbde800","Type":"ContainerStarted","Data":"e89d17abd174994dd48c517810b03febd8b7f08b66006598cb3d27717c92abc5"}
Sep 11 03:58:03.031084 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:03.031052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b724s" event={"ID":"624a6612-1d12-4e50-ba0d-07fdbdbde800","Type":"ContainerStarted","Data":"d7c56cc84f34249acdc72b5fd0cba43d1d3d7f4946f01e946f89518aa95f61e3"}
Sep 11 03:58:03.045313 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:03.045268    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-b724s" podStartSLOduration=1.203924817 podStartE2EDuration="3.045250927s" totalImagesPullingTime="1.84132611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:03.043734497 +0000 UTC m=+475.666127545" watchObservedRunningTime="2026-09-11 03:58:03.045250927 +0000 UTC m=+475.667643954"
Sep 11 03:58:03.684563 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:03.684529    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b724s"]
Sep 11 03:58:05.027987 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:05.027951    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:05.040724 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:05.040674    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-b724s" podUID="624a6612-1d12-4e50-ba0d-07fdbdbde800" containerName="authorino" containerID="cri-o://d7c56cc84f34249acdc72b5fd0cba43d1d3d7f4946f01e946f89518aa95f61e3" gracePeriod=30
Sep 11 03:58:05.095728 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:05.095694    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:05.281282 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:05.281226    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b724s"
Sep 11 03:58:05.386366 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:05.386336    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zntld\" (UniqueName: \"kubernetes.io/projected/624a6612-1d12-4e50-ba0d-07fdbdbde800-kube-api-access-zntld\") pod \"624a6612-1d12-4e50-ba0d-07fdbdbde800\" (UID: \"624a6612-1d12-4e50-ba0d-07fdbdbde800\") "
Sep 11 03:58:05.388810 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:05.388777    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624a6612-1d12-4e50-ba0d-07fdbdbde800-kube-api-access-zntld" (OuterVolumeSpecName: "kube-api-access-zntld") pod "624a6612-1d12-4e50-ba0d-07fdbdbde800" (UID: "624a6612-1d12-4e50-ba0d-07fdbdbde800"). InnerVolumeSpecName "kube-api-access-zntld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:58:05.487659 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:05.487630    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zntld\" (UniqueName: \"kubernetes.io/projected/624a6612-1d12-4e50-ba0d-07fdbdbde800-kube-api-access-zntld\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:58:06.046488 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:06.046457    2581 generic.go:356] "Generic (PLEG): container finished" podID="624a6612-1d12-4e50-ba0d-07fdbdbde800" containerID="d7c56cc84f34249acdc72b5fd0cba43d1d3d7f4946f01e946f89518aa95f61e3" exitCode=0
Sep 11 03:58:06.046895 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:06.046493    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b724s"
Sep 11 03:58:06.046895 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:06.046515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b724s" event={"ID":"624a6612-1d12-4e50-ba0d-07fdbdbde800","Type":"ContainerDied","Data":"d7c56cc84f34249acdc72b5fd0cba43d1d3d7f4946f01e946f89518aa95f61e3"}
Sep 11 03:58:06.046895 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:06.046545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b724s" event={"ID":"624a6612-1d12-4e50-ba0d-07fdbdbde800","Type":"ContainerDied","Data":"e89d17abd174994dd48c517810b03febd8b7f08b66006598cb3d27717c92abc5"}
Sep 11 03:58:06.046895 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:06.046561    2581 scope.go:117] "RemoveContainer" containerID="d7c56cc84f34249acdc72b5fd0cba43d1d3d7f4946f01e946f89518aa95f61e3"
Sep 11 03:58:06.054681 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:06.054663    2581 scope.go:117] "RemoveContainer" containerID="d7c56cc84f34249acdc72b5fd0cba43d1d3d7f4946f01e946f89518aa95f61e3"
Sep 11 03:58:06.054936 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:58:06.054918    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c56cc84f34249acdc72b5fd0cba43d1d3d7f4946f01e946f89518aa95f61e3\": container with ID starting with d7c56cc84f34249acdc72b5fd0cba43d1d3d7f4946f01e946f89518aa95f61e3 not found: ID does not exist" containerID="d7c56cc84f34249acdc72b5fd0cba43d1d3d7f4946f01e946f89518aa95f61e3"
Sep 11 03:58:06.054994 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:06.054949    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c56cc84f34249acdc72b5fd0cba43d1d3d7f4946f01e946f89518aa95f61e3"} err="failed to get container status \"d7c56cc84f34249acdc72b5fd0cba43d1d3d7f4946f01e946f89518aa95f61e3\": rpc error: code = NotFound desc = could not find container \"d7c56cc84f34249acdc72b5fd0cba43d1d3d7f4946f01e946f89518aa95f61e3\": container with ID starting with d7c56cc84f34249acdc72b5fd0cba43d1d3d7f4946f01e946f89518aa95f61e3 not found: ID does not exist"
Sep 11 03:58:06.063071 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:06.063048    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b724s"]
Sep 11 03:58:06.067069 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:06.067047    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b724s"]
Sep 11 03:58:07.479673 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:07.479640    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:07.534118 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:07.534085    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:07.861279 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:07.861254    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="624a6612-1d12-4e50-ba0d-07fdbdbde800" path="/var/lib/kubelet/pods/624a6612-1d12-4e50-ba0d-07fdbdbde800/volumes"
Sep 11 03:58:15.642945 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:15.642902    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:15.743282 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:15.743256    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:16.110483 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:16.110449    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:29.463648 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.463604    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:29.677097 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.677061    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-8szvl"]
Sep 11 03:58:29.677513 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.677497    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="624a6612-1d12-4e50-ba0d-07fdbdbde800" containerName="authorino"
Sep 11 03:58:29.677558 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.677515    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="624a6612-1d12-4e50-ba0d-07fdbdbde800" containerName="authorino"
Sep 11 03:58:29.677617 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.677609    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="624a6612-1d12-4e50-ba0d-07fdbdbde800" containerName="authorino"
Sep 11 03:58:29.683529 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.683509    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-8szvl"
Sep 11 03:58:29.686177 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.686158    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-r9bmw\""
Sep 11 03:58:29.691926 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.691908    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-8szvl"]
Sep 11 03:58:29.740521 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.740454    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:29.796529 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.796499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7blqn\" (UniqueName: \"kubernetes.io/projected/8373f252-02fb-4c84-8368-3b009af9611c-kube-api-access-7blqn\") pod \"authorino-557f876f69-8szvl\" (UID: \"8373f252-02fb-4c84-8368-3b009af9611c\") " pod="kuadrant-system/authorino-557f876f69-8szvl"
Sep 11 03:58:29.818648 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.818620    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-8szvl"]
Sep 11 03:58:29.818874 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:58:29.818857    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7blqn], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-8szvl" podUID="8373f252-02fb-4c84-8368-3b009af9611c"
Sep 11 03:58:29.847714 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.847678    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d4nfs"]
Sep 11 03:58:29.851931 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.851912    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d4nfs"
Sep 11 03:58:29.854309 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.854287    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 03:58:29.862380 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.862359    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d4nfs"]
Sep 11 03:58:29.896397 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.896375    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d4nfs"]
Sep 11 03:58:29.896689 ip-10-0-142-242 kubenswrapper[2581]: E0911 03:58:29.896670    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rk2f4 tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-rk2f4 tls-cert]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-d4nfs" podUID="4183450d-42f0-40d8-a085-fbe492cbdc56"
Sep 11 03:58:29.897058 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.897037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7blqn\" (UniqueName: \"kubernetes.io/projected/8373f252-02fb-4c84-8368-3b009af9611c-kube-api-access-7blqn\") pod \"authorino-557f876f69-8szvl\" (UID: \"8373f252-02fb-4c84-8368-3b009af9611c\") " pod="kuadrant-system/authorino-557f876f69-8szvl"
Sep 11 03:58:29.910067 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.910045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7blqn\" (UniqueName: \"kubernetes.io/projected/8373f252-02fb-4c84-8368-3b009af9611c-kube-api-access-7blqn\") pod \"authorino-557f876f69-8szvl\" (UID: \"8373f252-02fb-4c84-8368-3b009af9611c\") " pod="kuadrant-system/authorino-557f876f69-8szvl"
Sep 11 03:58:29.921033 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.921004    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-746b8f4bd-7xmvx"]
Sep 11 03:58:29.925309 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.925293    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-746b8f4bd-7xmvx"
Sep 11 03:58:29.934841 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.934823    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-746b8f4bd-7xmvx"]
Sep 11 03:58:29.998343 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.998270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk2f4\" (UniqueName: \"kubernetes.io/projected/4183450d-42f0-40d8-a085-fbe492cbdc56-kube-api-access-rk2f4\") pod \"authorino-6d4dfc649f-d4nfs\" (UID: \"4183450d-42f0-40d8-a085-fbe492cbdc56\") " pod="kuadrant-system/authorino-6d4dfc649f-d4nfs"
Sep 11 03:58:29.998343 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:29.998318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4183450d-42f0-40d8-a085-fbe492cbdc56-tls-cert\") pod \"authorino-6d4dfc649f-d4nfs\" (UID: \"4183450d-42f0-40d8-a085-fbe492cbdc56\") " pod="kuadrant-system/authorino-6d4dfc649f-d4nfs"
Sep 11 03:58:30.099328 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.099307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk2f4\" (UniqueName: \"kubernetes.io/projected/4183450d-42f0-40d8-a085-fbe492cbdc56-kube-api-access-rk2f4\") pod \"authorino-6d4dfc649f-d4nfs\" (UID: \"4183450d-42f0-40d8-a085-fbe492cbdc56\") " pod="kuadrant-system/authorino-6d4dfc649f-d4nfs"
Sep 11 03:58:30.099468 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.099334    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqh99\" (UniqueName: \"kubernetes.io/projected/dadb22b1-7f1b-41d0-ac64-1d0e32d32731-kube-api-access-nqh99\") pod \"authorino-746b8f4bd-7xmvx\" (UID: \"dadb22b1-7f1b-41d0-ac64-1d0e32d32731\") " pod="kuadrant-system/authorino-746b8f4bd-7xmvx"
Sep 11 03:58:30.099468 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.099354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dadb22b1-7f1b-41d0-ac64-1d0e32d32731-tls-cert\") pod \"authorino-746b8f4bd-7xmvx\" (UID: \"dadb22b1-7f1b-41d0-ac64-1d0e32d32731\") " pod="kuadrant-system/authorino-746b8f4bd-7xmvx"
Sep 11 03:58:30.099468 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.099391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4183450d-42f0-40d8-a085-fbe492cbdc56-tls-cert\") pod \"authorino-6d4dfc649f-d4nfs\" (UID: \"4183450d-42f0-40d8-a085-fbe492cbdc56\") " pod="kuadrant-system/authorino-6d4dfc649f-d4nfs"
Sep 11 03:58:30.101731 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.101709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4183450d-42f0-40d8-a085-fbe492cbdc56-tls-cert\") pod \"authorino-6d4dfc649f-d4nfs\" (UID: \"4183450d-42f0-40d8-a085-fbe492cbdc56\") " pod="kuadrant-system/authorino-6d4dfc649f-d4nfs"
Sep 11 03:58:30.107732 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.107711    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk2f4\" (UniqueName: \"kubernetes.io/projected/4183450d-42f0-40d8-a085-fbe492cbdc56-kube-api-access-rk2f4\") pod \"authorino-6d4dfc649f-d4nfs\" (UID: \"4183450d-42f0-40d8-a085-fbe492cbdc56\") " pod="kuadrant-system/authorino-6d4dfc649f-d4nfs"
Sep 11 03:58:30.156696 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.156673    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d4nfs"
Sep 11 03:58:30.156780 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.156677    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-8szvl"
Sep 11 03:58:30.164380 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.164364    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d4nfs"
Sep 11 03:58:30.167141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.167127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-8szvl"
Sep 11 03:58:30.199982 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.199955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqh99\" (UniqueName: \"kubernetes.io/projected/dadb22b1-7f1b-41d0-ac64-1d0e32d32731-kube-api-access-nqh99\") pod \"authorino-746b8f4bd-7xmvx\" (UID: \"dadb22b1-7f1b-41d0-ac64-1d0e32d32731\") " pod="kuadrant-system/authorino-746b8f4bd-7xmvx"
Sep 11 03:58:30.200125 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.199986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dadb22b1-7f1b-41d0-ac64-1d0e32d32731-tls-cert\") pod \"authorino-746b8f4bd-7xmvx\" (UID: \"dadb22b1-7f1b-41d0-ac64-1d0e32d32731\") " pod="kuadrant-system/authorino-746b8f4bd-7xmvx"
Sep 11 03:58:30.202170 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.202154    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dadb22b1-7f1b-41d0-ac64-1d0e32d32731-tls-cert\") pod \"authorino-746b8f4bd-7xmvx\" (UID: \"dadb22b1-7f1b-41d0-ac64-1d0e32d32731\") " pod="kuadrant-system/authorino-746b8f4bd-7xmvx"
Sep 11 03:58:30.207296 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.207279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqh99\" (UniqueName: \"kubernetes.io/projected/dadb22b1-7f1b-41d0-ac64-1d0e32d32731-kube-api-access-nqh99\") pod \"authorino-746b8f4bd-7xmvx\" (UID: \"dadb22b1-7f1b-41d0-ac64-1d0e32d32731\") " pod="kuadrant-system/authorino-746b8f4bd-7xmvx"
Sep 11 03:58:30.235146 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.235116    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-746b8f4bd-7xmvx"
Sep 11 03:58:30.301470 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.301440    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk2f4\" (UniqueName: \"kubernetes.io/projected/4183450d-42f0-40d8-a085-fbe492cbdc56-kube-api-access-rk2f4\") pod \"4183450d-42f0-40d8-a085-fbe492cbdc56\" (UID: \"4183450d-42f0-40d8-a085-fbe492cbdc56\") "
Sep 11 03:58:30.301592 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.301487    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7blqn\" (UniqueName: \"kubernetes.io/projected/8373f252-02fb-4c84-8368-3b009af9611c-kube-api-access-7blqn\") pod \"8373f252-02fb-4c84-8368-3b009af9611c\" (UID: \"8373f252-02fb-4c84-8368-3b009af9611c\") "
Sep 11 03:58:30.301658 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.301608    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4183450d-42f0-40d8-a085-fbe492cbdc56-tls-cert\") pod \"4183450d-42f0-40d8-a085-fbe492cbdc56\" (UID: \"4183450d-42f0-40d8-a085-fbe492cbdc56\") "
Sep 11 03:58:30.304337 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.304310    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4183450d-42f0-40d8-a085-fbe492cbdc56-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4183450d-42f0-40d8-a085-fbe492cbdc56" (UID: "4183450d-42f0-40d8-a085-fbe492cbdc56"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:58:30.304337 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.304328    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8373f252-02fb-4c84-8368-3b009af9611c-kube-api-access-7blqn" (OuterVolumeSpecName: "kube-api-access-7blqn") pod "8373f252-02fb-4c84-8368-3b009af9611c" (UID: "8373f252-02fb-4c84-8368-3b009af9611c"). InnerVolumeSpecName "kube-api-access-7blqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:58:30.304461 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.304305    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4183450d-42f0-40d8-a085-fbe492cbdc56-kube-api-access-rk2f4" (OuterVolumeSpecName: "kube-api-access-rk2f4") pod "4183450d-42f0-40d8-a085-fbe492cbdc56" (UID: "4183450d-42f0-40d8-a085-fbe492cbdc56"). InnerVolumeSpecName "kube-api-access-rk2f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:58:30.403036 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.402999    2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4183450d-42f0-40d8-a085-fbe492cbdc56-tls-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:58:30.403170 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.403042    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk2f4\" (UniqueName: \"kubernetes.io/projected/4183450d-42f0-40d8-a085-fbe492cbdc56-kube-api-access-rk2f4\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:58:30.403170 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.403054    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7blqn\" (UniqueName: \"kubernetes.io/projected/8373f252-02fb-4c84-8368-3b009af9611c-kube-api-access-7blqn\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 03:58:30.505723 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.505695    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-746b8f4bd-7xmvx"]
Sep 11 03:58:30.506877 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:58:30.506851    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddadb22b1_7f1b_41d0_ac64_1d0e32d32731.slice/crio-f529bddf182bdf0fbc438715419415eaa6bb3a6469ad5be07eae08d4016728b4 WatchSource:0}: Error finding container f529bddf182bdf0fbc438715419415eaa6bb3a6469ad5be07eae08d4016728b4: Status 404 returned error can't find the container with id f529bddf182bdf0fbc438715419415eaa6bb3a6469ad5be07eae08d4016728b4
Sep 11 03:58:30.564946 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:30.564920    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:31.162841 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:31.162800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-746b8f4bd-7xmvx" event={"ID":"dadb22b1-7f1b-41d0-ac64-1d0e32d32731","Type":"ContainerStarted","Data":"f529bddf182bdf0fbc438715419415eaa6bb3a6469ad5be07eae08d4016728b4"}
Sep 11 03:58:31.162841 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:31.162816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-8szvl"
Sep 11 03:58:31.163063 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:31.162821    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d4nfs"
Sep 11 03:58:31.195753 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:31.195720    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d4nfs"]
Sep 11 03:58:31.197516 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:31.197495    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d4nfs"]
Sep 11 03:58:31.220310 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:31.220269    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-8szvl"]
Sep 11 03:58:31.223272 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:31.223250    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-8szvl"]
Sep 11 03:58:31.861821 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:31.861742    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4183450d-42f0-40d8-a085-fbe492cbdc56" path="/var/lib/kubelet/pods/4183450d-42f0-40d8-a085-fbe492cbdc56/volumes"
Sep 11 03:58:31.862192 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:31.861963    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8373f252-02fb-4c84-8368-3b009af9611c" path="/var/lib/kubelet/pods/8373f252-02fb-4c84-8368-3b009af9611c/volumes"
Sep 11 03:58:32.175210 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:32.175124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-746b8f4bd-7xmvx" event={"ID":"dadb22b1-7f1b-41d0-ac64-1d0e32d32731","Type":"ContainerStarted","Data":"f6c8ad29a60bf3c59dd7a101e4c612d24e3143fb4f043ab91470bcad61cb8a1e"}
Sep 11 03:58:32.190557 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:32.190509    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-746b8f4bd-7xmvx" podStartSLOduration=2.13282074 podStartE2EDuration="3.190495857s" totalImagesPullingTime="1.057675117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:32.189278461 +0000 UTC m=+504.811671480" watchObservedRunningTime="2026-09-11 03:58:32.190495857 +0000 UTC m=+504.812888887"
Sep 11 03:58:36.821740 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:36.821705    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:36.898080 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:36.898016    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:36.973581 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:36.973547    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:37.096367 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:37.096280    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:37.378631 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:37.378534    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:40.738875 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:40.738838    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:40.836135 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:40.836087    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:46.706993 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:46.706954    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:58:46.811334 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:58:46.811295    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:05.838847 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:05.838766    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:05.949668 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:05.949615    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:12.341986 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:12.341948    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:12.432007 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:12.431961    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:13.831423 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:13.831390    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:13.931874 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:13.931842    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:14.117791 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:14.117705    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:14.303402 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:14.303362    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:14.401267 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:14.401187    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:14.474091 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:14.474056    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:25.982647 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:25.982607    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:26.082388 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:26.082357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:46.835411 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:46.835378    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:58.496517 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.496478    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"]
Sep 11 03:59:58.499267 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.499246    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.503158 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.503135    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 03:59:58.503291 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.503136    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 03:59:58.503291 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.503144    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 03:59:58.503291 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.503241    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jkjzv\""
Sep 11 03:59:58.510924 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.510901    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"]
Sep 11 03:59:58.535529 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.535495    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 03:59:58.675098 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.675067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eae38e05-6618-4768-9340-5e4a387551b9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.675265 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.675111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr7c4\" (UniqueName: \"kubernetes.io/projected/eae38e05-6618-4768-9340-5e4a387551b9-kube-api-access-dr7c4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.675265 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.675138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae38e05-6618-4768-9340-5e4a387551b9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.675265 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.675214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eae38e05-6618-4768-9340-5e4a387551b9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.675265 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.675256    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eae38e05-6618-4768-9340-5e4a387551b9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.675397 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.675293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae38e05-6618-4768-9340-5e4a387551b9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.775934 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.775855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae38e05-6618-4768-9340-5e4a387551b9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.776107 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.775950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eae38e05-6618-4768-9340-5e4a387551b9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.776107 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.775996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr7c4\" (UniqueName: \"kubernetes.io/projected/eae38e05-6618-4768-9340-5e4a387551b9-kube-api-access-dr7c4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.776107 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.776043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae38e05-6618-4768-9340-5e4a387551b9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.776283 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.776110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eae38e05-6618-4768-9340-5e4a387551b9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.776283 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.776148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eae38e05-6618-4768-9340-5e4a387551b9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.776390 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.776321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae38e05-6618-4768-9340-5e4a387551b9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.776457 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.776431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eae38e05-6618-4768-9340-5e4a387551b9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.776545 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.776525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eae38e05-6618-4768-9340-5e4a387551b9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.778141 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.778115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eae38e05-6618-4768-9340-5e4a387551b9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.778418 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.778399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae38e05-6618-4768-9340-5e4a387551b9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.785036 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.785003    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr7c4\" (UniqueName: \"kubernetes.io/projected/eae38e05-6618-4768-9340-5e4a387551b9-kube-api-access-dr7c4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tqdk7\" (UID: \"eae38e05-6618-4768-9340-5e4a387551b9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:58.810665 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:58.810640    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 03:59:59.094582 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:59.094554    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"]
Sep 11 03:59:59.097035 ip-10-0-142-242 kubenswrapper[2581]: W0911 03:59:59.096989    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae38e05_6618_4768_9340_5e4a387551b9.slice/crio-a85b4831a115773a3d09c8bb4da46430e40bcb3835130f059f74207cbc9a5069 WatchSource:0}: Error finding container a85b4831a115773a3d09c8bb4da46430e40bcb3835130f059f74207cbc9a5069: Status 404 returned error can't find the container with id a85b4831a115773a3d09c8bb4da46430e40bcb3835130f059f74207cbc9a5069
Sep 11 03:59:59.595461 ip-10-0-142-242 kubenswrapper[2581]: I0911 03:59:59.595387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7" event={"ID":"eae38e05-6618-4768-9340-5e4a387551b9","Type":"ContainerStarted","Data":"a85b4831a115773a3d09c8bb4da46430e40bcb3835130f059f74207cbc9a5069"}
Sep 11 04:00:00.139346 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:00.139314    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818320-b9pmz"]
Sep 11 04:00:00.142317 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:00.142296    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-b9pmz"
Sep 11 04:00:00.145037 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:00.145000    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbtxz\""
Sep 11 04:00:00.148605 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:00.148579    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818320-b9pmz"]
Sep 11 04:00:00.156811 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:00.156788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-b9pmz"
Sep 11 04:00:00.445409 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:00.445379    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818320-b9pmz"]
Sep 11 04:00:00.447013 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:00:00.446979    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96d5a3e_44ae_4d7b_920f_85f75446e414.slice/crio-d0aaf56cf1721e2445086518a08875c090adbb783eb6d3cef7faa2c8cd41202d WatchSource:0}: Error finding container d0aaf56cf1721e2445086518a08875c090adbb783eb6d3cef7faa2c8cd41202d: Status 404 returned error can't find the container with id d0aaf56cf1721e2445086518a08875c090adbb783eb6d3cef7faa2c8cd41202d
Sep 11 04:00:00.602769 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:00.602728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7" event={"ID":"eae38e05-6618-4768-9340-5e4a387551b9","Type":"ContainerStarted","Data":"34b82cbd20df44fecc1ef63f05d6812ee2edc30da69d64c0f134ca940ee51ab5"}
Sep 11 04:00:00.604077 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:00.604044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-b9pmz" event={"ID":"a96d5a3e-44ae-4d7b-920f-85f75446e414","Type":"ContainerStarted","Data":"d0aaf56cf1721e2445086518a08875c090adbb783eb6d3cef7faa2c8cd41202d"}
Sep 11 04:00:00.620310 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:00.620266    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7" podStartSLOduration=1.5058203350000001 podStartE2EDuration="2.620252311s" totalImagesPullingTime="1.114431976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:00.619250371 +0000 UTC m=+593.241643430" watchObservedRunningTime="2026-09-11 04:00:00.620252311 +0000 UTC m=+593.242645338"
Sep 11 04:00:03.058141 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:03.058107    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:00:04.627421 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:04.627331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-b9pmz" event={"ID":"a96d5a3e-44ae-4d7b-920f-85f75446e414","Type":"ContainerStarted","Data":"4b178761cacfbde9888d36c8ca9b6e96b8c5bfcc87d8896b71cbcca68daa85cc"}
Sep 11 04:00:04.658167 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:04.658108    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-b9pmz" podStartSLOduration=1.063619167 podStartE2EDuration="4.658092945s" totalImagesPullingTime="3.594473778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:04.655838009 +0000 UTC m=+597.278231055" watchObservedRunningTime="2026-09-11 04:00:04.658092945 +0000 UTC m=+597.280485971"
Sep 11 04:00:05.639305 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:05.639263    2581 generic.go:356] "Generic (PLEG): container finished" podID="a96d5a3e-44ae-4d7b-920f-85f75446e414" containerID="4b178761cacfbde9888d36c8ca9b6e96b8c5bfcc87d8896b71cbcca68daa85cc" exitCode=0
Sep 11 04:00:05.639717 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:05.639310    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-b9pmz" event={"ID":"a96d5a3e-44ae-4d7b-920f-85f75446e414","Type":"ContainerDied","Data":"4b178761cacfbde9888d36c8ca9b6e96b8c5bfcc87d8896b71cbcca68daa85cc"}
Sep 11 04:00:06.778863 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:06.778839    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-b9pmz"
Sep 11 04:00:07.651100 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:07.651060    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-b9pmz" event={"ID":"a96d5a3e-44ae-4d7b-920f-85f75446e414","Type":"ContainerDied","Data":"d0aaf56cf1721e2445086518a08875c090adbb783eb6d3cef7faa2c8cd41202d"}
Sep 11 04:00:07.651258 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:07.651118    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0aaf56cf1721e2445086518a08875c090adbb783eb6d3cef7faa2c8cd41202d"
Sep 11 04:00:07.651258 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:07.651073    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-b9pmz"
Sep 11 04:00:08.810992 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:08.810954    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 04:00:08.811467 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:08.811049    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 04:00:08.823574 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:08.823547    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 04:00:09.678219 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:09.678189    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tqdk7"
Sep 11 04:00:14.535855 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:14.535818    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:00:27.306246 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.306211    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"]
Sep 11 04:00:27.306802 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.306776    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96d5a3e-44ae-4d7b-920f-85f75446e414" containerName="cleanup"
Sep 11 04:00:27.306802 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.306801    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96d5a3e-44ae-4d7b-920f-85f75446e414" containerName="cleanup"
Sep 11 04:00:27.306991 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.306938    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96d5a3e-44ae-4d7b-920f-85f75446e414" containerName="cleanup"
Sep 11 04:00:27.309248 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.309227    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.311820 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.311800    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 04:00:27.320313 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.320289    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"]
Sep 11 04:00:27.343039 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.342980    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:00:27.440348 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.440312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62213da7-5243-4289-aa68-9d57ca9f9e93-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.440348 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.440348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62213da7-5243-4289-aa68-9d57ca9f9e93-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.440606 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.440369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62213da7-5243-4289-aa68-9d57ca9f9e93-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.440606 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.440488    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pf5m\" (UniqueName: \"kubernetes.io/projected/62213da7-5243-4289-aa68-9d57ca9f9e93-kube-api-access-2pf5m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.440606 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.440553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62213da7-5243-4289-aa68-9d57ca9f9e93-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.440606 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.440583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62213da7-5243-4289-aa68-9d57ca9f9e93-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.541811 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.541772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pf5m\" (UniqueName: \"kubernetes.io/projected/62213da7-5243-4289-aa68-9d57ca9f9e93-kube-api-access-2pf5m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.541979 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.541825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62213da7-5243-4289-aa68-9d57ca9f9e93-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.541979 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.541946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62213da7-5243-4289-aa68-9d57ca9f9e93-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.542136 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.542092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62213da7-5243-4289-aa68-9d57ca9f9e93-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.542136 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.542124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62213da7-5243-4289-aa68-9d57ca9f9e93-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.542239 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.542168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62213da7-5243-4289-aa68-9d57ca9f9e93-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.542239 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.542208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62213da7-5243-4289-aa68-9d57ca9f9e93-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.542327 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.542260    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62213da7-5243-4289-aa68-9d57ca9f9e93-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.542443 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.542424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62213da7-5243-4289-aa68-9d57ca9f9e93-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.544332 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.544312    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62213da7-5243-4289-aa68-9d57ca9f9e93-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.544561 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.544546    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62213da7-5243-4289-aa68-9d57ca9f9e93-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.561630 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.561580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pf5m\" (UniqueName: \"kubernetes.io/projected/62213da7-5243-4289-aa68-9d57ca9f9e93-kube-api-access-2pf5m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm\" (UID: \"62213da7-5243-4289-aa68-9d57ca9f9e93\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.623312 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.623284    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:27.894877 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.894815    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"]
Sep 11 04:00:27.897449 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:00:27.897423    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62213da7_5243_4289_aa68_9d57ca9f9e93.slice/crio-d5419a5c6311d517cdcc80fbf390a85cca930d6811c1c253cada51075d3b763e WatchSource:0}: Error finding container d5419a5c6311d517cdcc80fbf390a85cca930d6811c1c253cada51075d3b763e: Status 404 returned error can't find the container with id d5419a5c6311d517cdcc80fbf390a85cca930d6811c1c253cada51075d3b763e
Sep 11 04:00:27.898805 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:27.898788    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 04:00:28.754102 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:28.754067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm" event={"ID":"62213da7-5243-4289-aa68-9d57ca9f9e93","Type":"ContainerStarted","Data":"d5419a5c6311d517cdcc80fbf390a85cca930d6811c1c253cada51075d3b763e"}
Sep 11 04:00:31.828485 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:31.828371    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:00:32.775610 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:32.775529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm" event={"ID":"62213da7-5243-4289-aa68-9d57ca9f9e93","Type":"ContainerStarted","Data":"6ceb0b446e91a92afaa1353ef5236d6791f88970a127398c3993eb7bbd1b1dfb"}
Sep 11 04:00:32.794920 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:32.794870    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm" podStartSLOduration=1.234191178 podStartE2EDuration="5.794854378s" totalImagesPullingTime="4.5606632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:32.791961739 +0000 UTC m=+625.414354789" watchObservedRunningTime="2026-09-11 04:00:32.794854378 +0000 UTC m=+625.417247404"
Sep 11 04:00:33.416449 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:33.416415    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:00:37.623447 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:37.623411    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:37.623893 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:37.623481    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:37.636163 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:37.636134    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:00:37.812373 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:00:37.812343    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm"
Sep 11 04:01:27.127843 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:27.127808    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-756b658b9-xz8t6"]
Sep 11 04:01:27.130431 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:27.130411    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-756b658b9-xz8t6"
Sep 11 04:01:27.140121 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:27.140097    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-756b658b9-xz8t6"]
Sep 11 04:01:27.180614 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:27.180586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99b7a60f-140d-45c1-a886-ec51b9fc1375-tls-cert\") pod \"authorino-756b658b9-xz8t6\" (UID: \"99b7a60f-140d-45c1-a886-ec51b9fc1375\") " pod="kuadrant-system/authorino-756b658b9-xz8t6"
Sep 11 04:01:27.180718 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:27.180669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glxbw\" (UniqueName: \"kubernetes.io/projected/99b7a60f-140d-45c1-a886-ec51b9fc1375-kube-api-access-glxbw\") pod \"authorino-756b658b9-xz8t6\" (UID: \"99b7a60f-140d-45c1-a886-ec51b9fc1375\") " pod="kuadrant-system/authorino-756b658b9-xz8t6"
Sep 11 04:01:27.281077 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:27.281048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glxbw\" (UniqueName: \"kubernetes.io/projected/99b7a60f-140d-45c1-a886-ec51b9fc1375-kube-api-access-glxbw\") pod \"authorino-756b658b9-xz8t6\" (UID: \"99b7a60f-140d-45c1-a886-ec51b9fc1375\") " pod="kuadrant-system/authorino-756b658b9-xz8t6"
Sep 11 04:01:27.281231 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:27.281117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99b7a60f-140d-45c1-a886-ec51b9fc1375-tls-cert\") pod \"authorino-756b658b9-xz8t6\" (UID: \"99b7a60f-140d-45c1-a886-ec51b9fc1375\") " pod="kuadrant-system/authorino-756b658b9-xz8t6"
Sep 11 04:01:27.283460 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:27.283437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99b7a60f-140d-45c1-a886-ec51b9fc1375-tls-cert\") pod \"authorino-756b658b9-xz8t6\" (UID: \"99b7a60f-140d-45c1-a886-ec51b9fc1375\") " pod="kuadrant-system/authorino-756b658b9-xz8t6"
Sep 11 04:01:27.296403 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:27.296378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glxbw\" (UniqueName: \"kubernetes.io/projected/99b7a60f-140d-45c1-a886-ec51b9fc1375-kube-api-access-glxbw\") pod \"authorino-756b658b9-xz8t6\" (UID: \"99b7a60f-140d-45c1-a886-ec51b9fc1375\") " pod="kuadrant-system/authorino-756b658b9-xz8t6"
Sep 11 04:01:27.440225 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:27.440141    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-756b658b9-xz8t6"
Sep 11 04:01:27.710966 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:27.710941    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-756b658b9-xz8t6"]
Sep 11 04:01:27.713806 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:01:27.713778    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99b7a60f_140d_45c1_a886_ec51b9fc1375.slice/crio-b3f8c49e8ffa2b7ba3f5035e607fa850218411d0a07503cdf714d5d98d8875f3 WatchSource:0}: Error finding container b3f8c49e8ffa2b7ba3f5035e607fa850218411d0a07503cdf714d5d98d8875f3: Status 404 returned error can't find the container with id b3f8c49e8ffa2b7ba3f5035e607fa850218411d0a07503cdf714d5d98d8875f3
Sep 11 04:01:28.060145 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:28.060110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-756b658b9-xz8t6" event={"ID":"99b7a60f-140d-45c1-a886-ec51b9fc1375","Type":"ContainerStarted","Data":"b3f8c49e8ffa2b7ba3f5035e607fa850218411d0a07503cdf714d5d98d8875f3"}
Sep 11 04:01:29.067395 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:29.067348    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-756b658b9-xz8t6" event={"ID":"99b7a60f-140d-45c1-a886-ec51b9fc1375","Type":"ContainerStarted","Data":"19fd9b7b3e06bf5716ee9750bc33ff592ae4b09822c2fb713bdb5631fe7cfd8a"}
Sep 11 04:01:29.087206 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:29.087147    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-756b658b9-xz8t6" podStartSLOduration=1.505108008 podStartE2EDuration="2.08709739s" totalImagesPullingTime="581.989382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:01:29.086429772 +0000 UTC m=+681.708822802" watchObservedRunningTime="2026-09-11 04:01:29.08709739 +0000 UTC m=+681.709490417"
Sep 11 04:01:29.125905 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:29.125868    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-746b8f4bd-7xmvx"]
Sep 11 04:01:29.126139 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:29.126105    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-746b8f4bd-7xmvx" podUID="dadb22b1-7f1b-41d0-ac64-1d0e32d32731" containerName="authorino" containerID="cri-o://f6c8ad29a60bf3c59dd7a101e4c612d24e3143fb4f043ab91470bcad61cb8a1e" gracePeriod=30
Sep 11 04:01:29.376952 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:29.376915    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-746b8f4bd-7xmvx"
Sep 11 04:01:29.403818 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:29.403786    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dadb22b1-7f1b-41d0-ac64-1d0e32d32731-tls-cert\") pod \"dadb22b1-7f1b-41d0-ac64-1d0e32d32731\" (UID: \"dadb22b1-7f1b-41d0-ac64-1d0e32d32731\") "
Sep 11 04:01:29.403986 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:29.403897    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqh99\" (UniqueName: \"kubernetes.io/projected/dadb22b1-7f1b-41d0-ac64-1d0e32d32731-kube-api-access-nqh99\") pod \"dadb22b1-7f1b-41d0-ac64-1d0e32d32731\" (UID: \"dadb22b1-7f1b-41d0-ac64-1d0e32d32731\") "
Sep 11 04:01:29.406245 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:29.406214    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadb22b1-7f1b-41d0-ac64-1d0e32d32731-kube-api-access-nqh99" (OuterVolumeSpecName: "kube-api-access-nqh99") pod "dadb22b1-7f1b-41d0-ac64-1d0e32d32731" (UID: "dadb22b1-7f1b-41d0-ac64-1d0e32d32731"). InnerVolumeSpecName "kube-api-access-nqh99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:01:29.416603 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:29.416569    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dadb22b1-7f1b-41d0-ac64-1d0e32d32731-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "dadb22b1-7f1b-41d0-ac64-1d0e32d32731" (UID: "dadb22b1-7f1b-41d0-ac64-1d0e32d32731"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:01:29.505777 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:29.505745    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqh99\" (UniqueName: \"kubernetes.io/projected/dadb22b1-7f1b-41d0-ac64-1d0e32d32731-kube-api-access-nqh99\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:01:29.505777 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:29.505772    2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dadb22b1-7f1b-41d0-ac64-1d0e32d32731-tls-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:01:29.630852 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:29.630757    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:29.724943 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:29.724901    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:30.074044 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:30.073980    2581 generic.go:356] "Generic (PLEG): container finished" podID="dadb22b1-7f1b-41d0-ac64-1d0e32d32731" containerID="f6c8ad29a60bf3c59dd7a101e4c612d24e3143fb4f043ab91470bcad61cb8a1e" exitCode=0
Sep 11 04:01:30.074555 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:30.074067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-746b8f4bd-7xmvx" event={"ID":"dadb22b1-7f1b-41d0-ac64-1d0e32d32731","Type":"ContainerDied","Data":"f6c8ad29a60bf3c59dd7a101e4c612d24e3143fb4f043ab91470bcad61cb8a1e"}
Sep 11 04:01:30.074555 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:30.074131    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-746b8f4bd-7xmvx" event={"ID":"dadb22b1-7f1b-41d0-ac64-1d0e32d32731","Type":"ContainerDied","Data":"f529bddf182bdf0fbc438715419415eaa6bb3a6469ad5be07eae08d4016728b4"}
Sep 11 04:01:30.074555 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:30.074158    2581 scope.go:117] "RemoveContainer" containerID="f6c8ad29a60bf3c59dd7a101e4c612d24e3143fb4f043ab91470bcad61cb8a1e"
Sep 11 04:01:30.074555 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:30.074292    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-746b8f4bd-7xmvx"
Sep 11 04:01:30.085735 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:30.085710    2581 scope.go:117] "RemoveContainer" containerID="f6c8ad29a60bf3c59dd7a101e4c612d24e3143fb4f043ab91470bcad61cb8a1e"
Sep 11 04:01:30.086042 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:01:30.086004    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c8ad29a60bf3c59dd7a101e4c612d24e3143fb4f043ab91470bcad61cb8a1e\": container with ID starting with f6c8ad29a60bf3c59dd7a101e4c612d24e3143fb4f043ab91470bcad61cb8a1e not found: ID does not exist" containerID="f6c8ad29a60bf3c59dd7a101e4c612d24e3143fb4f043ab91470bcad61cb8a1e"
Sep 11 04:01:30.086150 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:30.086044    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c8ad29a60bf3c59dd7a101e4c612d24e3143fb4f043ab91470bcad61cb8a1e"} err="failed to get container status \"f6c8ad29a60bf3c59dd7a101e4c612d24e3143fb4f043ab91470bcad61cb8a1e\": rpc error: code = NotFound desc = could not find container \"f6c8ad29a60bf3c59dd7a101e4c612d24e3143fb4f043ab91470bcad61cb8a1e\": container with ID starting with f6c8ad29a60bf3c59dd7a101e4c612d24e3143fb4f043ab91470bcad61cb8a1e not found: ID does not exist"
Sep 11 04:01:30.091916 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:30.091892    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-746b8f4bd-7xmvx"]
Sep 11 04:01:30.098649 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:30.098628    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-746b8f4bd-7xmvx"]
Sep 11 04:01:30.644654 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:30.644614    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:30.746650 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:30.746616    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:31.861061 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:31.861010    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dadb22b1-7f1b-41d0-ac64-1d0e32d32731" path="/var/lib/kubelet/pods/dadb22b1-7f1b-41d0-ac64-1d0e32d32731/volumes"
Sep 11 04:01:45.647155 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:45.647120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:45.786792 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:45.786760    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:45.937301 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:45.937225    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:46.438742 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:46.438704    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:46.995425 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:46.995388    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:47.095615 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:47.095580    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:47.478609 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:47.478575    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:49.219312 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:49.219267    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:49.315865 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:49.315825    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:50.329228 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.329194    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"]
Sep 11 04:01:50.329875 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.329846    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dadb22b1-7f1b-41d0-ac64-1d0e32d32731" containerName="authorino"
Sep 11 04:01:50.329875 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.329875    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dadb22b1-7f1b-41d0-ac64-1d0e32d32731" containerName="authorino"
Sep 11 04:01:50.330107 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.329968    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dadb22b1-7f1b-41d0-ac64-1d0e32d32731" containerName="authorino"
Sep 11 04:01:50.332918 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.332895    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:50.335934 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.335910    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-00a938-serving-cert\""
Sep 11 04:01:50.336229 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.336184    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 04:01:50.337135 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.337112    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:01:50.337135 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.337126    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wb8bv\""
Sep 11 04:01:50.337322 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.337218    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 04:01:50.348786 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.348763    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"]
Sep 11 04:01:50.377381 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.377346    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:50.403531 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.403490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-metrics-certs\") pod \"maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:50.403686 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.403539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-maas-api-tls\") pod \"maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:50.403686 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.403606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t6k7\" (UniqueName: \"kubernetes.io/projected/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-kube-api-access-7t6k7\") pod \"maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:50.403812 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.403713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:50.459368 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.459336    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:50.504848 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.504822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-metrics-certs\") pod \"maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:50.504995 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.504859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-maas-api-tls\") pod \"maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:50.504995 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.504885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t6k7\" (UniqueName: \"kubernetes.io/projected/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-kube-api-access-7t6k7\") pod \"maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:50.504995 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.504954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:50.505661 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.505635    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:50.507388 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.507365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-maas-api-tls\") pod \"maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:50.507469 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.507365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-metrics-certs\") pod \"maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:50.520666 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.520636    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t6k7\" (UniqueName: \"kubernetes.io/projected/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-kube-api-access-7t6k7\") pod \"maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:50.575754 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.575722    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:50.644939 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.644855    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:50.917772 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:50.917740    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"]
Sep 11 04:01:50.919455 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:01:50.919430    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060eb9dd_78ae_4b3e_be8e_1d83c2b48b8a.slice/crio-c3c54a7e724df85da00668a828d2013eb01d13a458f52447115643b80bcc1af8 WatchSource:0}: Error finding container c3c54a7e724df85da00668a828d2013eb01d13a458f52447115643b80bcc1af8: Status 404 returned error can't find the container with id c3c54a7e724df85da00668a828d2013eb01d13a458f52447115643b80bcc1af8
Sep 11 04:01:51.013235 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:51.013200    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:51.117937 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:51.117903    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:51.188101 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:51.187344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv" event={"ID":"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a","Type":"ContainerStarted","Data":"c3c54a7e724df85da00668a828d2013eb01d13a458f52447115643b80bcc1af8"}
Sep 11 04:01:51.790603 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:51.790568    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"]
Sep 11 04:01:51.793409 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:51.793389    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:51.796121 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:51.796101    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-92d75d22-kserve-self-signed-certs\""
Sep 11 04:01:51.803583 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:51.803560    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"]
Sep 11 04:01:51.920478 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:51.920112    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-tmp-dir\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:51.920478 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:51.920170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-home\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:51.920478 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:51.920218    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-model-cache\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:51.920478 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:51.920275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727c386d-ed7f-4d43-aceb-32109e392716-tls-certs\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:51.920478 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:51.920359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scr75\" (UniqueName: \"kubernetes.io/projected/727c386d-ed7f-4d43-aceb-32109e392716-kube-api-access-scr75\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:51.920478 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:51.920386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-dshm\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:52.022072 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.021308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scr75\" (UniqueName: \"kubernetes.io/projected/727c386d-ed7f-4d43-aceb-32109e392716-kube-api-access-scr75\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:52.022072 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.021359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-dshm\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:52.022072 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.021400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-tmp-dir\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:52.022072 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.021437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-home\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:52.022072 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.021486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-model-cache\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:52.022072 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.021549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727c386d-ed7f-4d43-aceb-32109e392716-tls-certs\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:52.022072 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.021981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-home\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:52.022574 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.022303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-tmp-dir\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:52.022574 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.022438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-model-cache\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:52.024162 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.024116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-dshm\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:52.027720 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.027696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727c386d-ed7f-4d43-aceb-32109e392716-tls-certs\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:52.032290 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.032263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scr75\" (UniqueName: \"kubernetes.io/projected/727c386d-ed7f-4d43-aceb-32109e392716-kube-api-access-scr75\") pod \"e2e-conflict-a-92d75d22-kserve-857845995b-5twpv\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") " pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:52.105978 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.105946    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:01:52.197236 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.197202    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:52.404255 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:52.404226    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"]
Sep 11 04:01:53.250476 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:01:53.250447    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod727c386d_ed7f_4d43_aceb_32109e392716.slice/crio-6088b982d96eec235addf1a16926814aff95d830ec4a6e340ba0c5456ca727be WatchSource:0}: Error finding container 6088b982d96eec235addf1a16926814aff95d830ec4a6e340ba0c5456ca727be: Status 404 returned error can't find the container with id 6088b982d96eec235addf1a16926814aff95d830ec4a6e340ba0c5456ca727be
Sep 11 04:01:53.706325 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:53.706292    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:53.813848 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:53.813813    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:53.919382 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:53.919349    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:54.068059 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:54.067998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:54.206335 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:54.206297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv" event={"ID":"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a","Type":"ContainerStarted","Data":"2edc2314d58f4d226cd013f47ad51496b10a4742ca7d99427f5f6466bc25c5fa"}
Sep 11 04:01:54.206510 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:54.206485    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:01:54.207904 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:54.207879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv" event={"ID":"727c386d-ed7f-4d43-aceb-32109e392716","Type":"ContainerStarted","Data":"9ee5f8089268329e87ef1f01210544d4af482d192ae7bc8369674ee485b4fee4"}
Sep 11 04:01:54.208009 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:54.207913    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv" event={"ID":"727c386d-ed7f-4d43-aceb-32109e392716","Type":"ContainerStarted","Data":"6088b982d96eec235addf1a16926814aff95d830ec4a6e340ba0c5456ca727be"}
Sep 11 04:01:54.232389 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:54.232339    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv" podStartSLOduration=1.868012088 podStartE2EDuration="4.232323224s" totalImagesPullingTime="2.364311136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:01:54.231875657 +0000 UTC m=+706.854268686" watchObservedRunningTime="2026-09-11 04:01:54.232323224 +0000 UTC m=+706.854716251"
Sep 11 04:01:54.259047 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:54.258969    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv" podStartSLOduration=3.258948567 podStartE2EDuration="3.258948567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:01:54.255425062 +0000 UTC m=+706.877818090" watchObservedRunningTime="2026-09-11 04:01:54.258948567 +0000 UTC m=+706.881341595"
Sep 11 04:01:54.268144 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:54.268111    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:54.627456 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:54.627417    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:54.795809 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:54.795777    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:56.699920 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:56.699882    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:56.818765 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:56.818727    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:57.781625 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:57.781571    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:57.940475 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:57.940437    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:58.156961 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:58.156933    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:58.256811 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:58.256768    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:58.506467 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:58.506385    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:58.876904 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:58.876860    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:58.964800 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:58.964759    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:59.403234 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:59.403202    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:01:59.527740 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:01:59.527702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:00.111958 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:00.111923    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:00.190281 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:00.190240    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:00.217850 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:00.217825    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:02:00.528077 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:00.527921    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:01.522815 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:01.522766    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:02.106194 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:02.106156    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:02:02.106387 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:02.106211    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:02:02.119453 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:02.119422    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:02:02.262496 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:02.262469    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:02:04.510468 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:04.510435    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:04.722423 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:04.722384    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:04.818826 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:04.818795    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:04.922195 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:04.922163    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:07.521600 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:07.521566    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:09.149147 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:09.149095    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:11.411910 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:11.411866    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:11.577405 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:11.577367    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:11.675424 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:11.675343    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:14.000730 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:14.000687    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:14.354664 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:14.354627    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:14.647795 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:14.647704    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:15.105818 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:15.105753    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:17.749710 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:17.749652    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:17.852375 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:17.852339    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:17.951727 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:17.951686    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:18.085770 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:18.085741    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:19.082199 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.082165    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"]
Sep 11 04:02:19.091376 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.091352    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:19.094673 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.094656    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5d685d-serving-cert\""
Sep 11 04:02:19.106884 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.106858    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"]
Sep 11 04:02:19.145858 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.145831    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:19.186169 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.186138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:19.186286 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.186174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-metrics-certs\") pod \"maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:19.186286 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.186263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9lsj\" (UniqueName: \"kubernetes.io/projected/2bdd0cfa-7899-4c06-a7c8-260406688630-kube-api-access-v9lsj\") pod \"maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:19.186359 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.186289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdd0cfa-7899-4c06-a7c8-260406688630-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:19.230780 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.230750    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:19.286731 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.286705    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:19.286849 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.286735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-metrics-certs\") pod \"maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:19.286849 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.286803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9lsj\" (UniqueName: \"kubernetes.io/projected/2bdd0cfa-7899-4c06-a7c8-260406688630-kube-api-access-v9lsj\") pod \"maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:19.286849 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.286834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdd0cfa-7899-4c06-a7c8-260406688630-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:19.286955 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:02:19.286846    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-serving-cert: secret "maas-api-e2e-inf-b-5d685d-serving-cert" not found
Sep 11 04:02:19.286955 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:02:19.286912    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-maas-api-tls podName:2bdd0cfa-7899-4c06-a7c8-260406688630 nodeName:}" failed. No retries permitted until 2026-09-11 04:02:19.786898375 +0000 UTC m=+732.409291380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-maas-api-tls") pod "maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz" (UID: "2bdd0cfa-7899-4c06-a7c8-260406688630") : secret "maas-api-e2e-inf-b-5d685d-serving-cert" not found
Sep 11 04:02:19.287533 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.287508    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdd0cfa-7899-4c06-a7c8-260406688630-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:19.289216 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.289199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-metrics-certs\") pod \"maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:19.298797 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.298777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9lsj\" (UniqueName: \"kubernetes.io/projected/2bdd0cfa-7899-4c06-a7c8-260406688630-kube-api-access-v9lsj\") pod \"maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:19.364998 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.364909    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:19.624946 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.624850    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:19.720587 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.720551    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:19.792168 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.792138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:19.794562 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:19.794538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:20.001474 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:20.001380    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:20.049983 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:20.049949    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:20.291953 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:20.291928    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"]
Sep 11 04:02:20.293804 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:02:20.293779    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdd0cfa_7899_4c06_a7c8_260406688630.slice/crio-8d441936a0cd03a0c913418b0f62ab4e07df7ef5f008dc5ea5f6abd76a548b44 WatchSource:0}: Error finding container 8d441936a0cd03a0c913418b0f62ab4e07df7ef5f008dc5ea5f6abd76a548b44: Status 404 returned error can't find the container with id 8d441936a0cd03a0c913418b0f62ab4e07df7ef5f008dc5ea5f6abd76a548b44
Sep 11 04:02:20.348899 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:20.348868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz" event={"ID":"2bdd0cfa-7899-4c06-a7c8-260406688630","Type":"ContainerStarted","Data":"8d441936a0cd03a0c913418b0f62ab4e07df7ef5f008dc5ea5f6abd76a548b44"}
Sep 11 04:02:21.131003 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:21.130967    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:21.355728 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:21.355693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz" event={"ID":"2bdd0cfa-7899-4c06-a7c8-260406688630","Type":"ContainerStarted","Data":"7873b7f7e68289eb5156d7f31254a72bce19f3d16cac87d56de18aaf1d963140"}
Sep 11 04:02:21.356134 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:21.355787    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:21.373423 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:21.373371    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz" podStartSLOduration=2.013736107 podStartE2EDuration="2.373356852s" totalImagesPullingTime="359.620745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:02:21.372384761 +0000 UTC m=+733.994777834" watchObservedRunningTime="2026-09-11 04:02:21.373356852 +0000 UTC m=+733.995749879"
Sep 11 04:02:21.862899 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:21.862866    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:22.769865 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:22.769811    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:23.662098 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:23.662059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:24.541105 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:24.541072    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:24.638649 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:24.638612    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:27.366361 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:27.366331    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:02:27.693560 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:27.693468    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:29.891139 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:29.891097    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:38.345716 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:38.345671    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:38.981436 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:38.981396    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:39.064541 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:39.064499    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:39.119470 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:39.119438    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:39.220994 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:39.220952    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:39.292058 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:39.291966    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:39.362841 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:39.362790    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:39.820835 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:39.820800    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:39.923643 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:39.923608    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:40.012984 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:40.012930    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:40.169543 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:40.169449    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:40.329366 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:40.329332    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:40.734615 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:40.734573    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:42.707412 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:42.707380    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:43.464211 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.464175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:43.566184 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.566141    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:43.703061 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.703013    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"]
Sep 11 04:02:43.707074 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.707053    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:43.709517 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.709495    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d3676176-serving-cert\""
Sep 11 04:02:43.718227 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.718202    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"]
Sep 11 04:02:43.724270 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.724246    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c593c7-be5a-4eef-94a2-5ea1091e93bd-openshift-service-ca\") pod \"maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:43.724363 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.724290    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dm6h\" (UniqueName: \"kubernetes.io/projected/94c593c7-be5a-4eef-94a2-5ea1091e93bd-kube-api-access-7dm6h\") pod \"maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:43.724363 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.724338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-maas-api-tls\") pod \"maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:43.724518 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.724481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-metrics-certs\") pod \"maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:43.738385 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.738350    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:43.824937 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.824908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-metrics-certs\") pod \"maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:43.825098 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.824962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c593c7-be5a-4eef-94a2-5ea1091e93bd-openshift-service-ca\") pod \"maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:43.825098 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.824985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dm6h\" (UniqueName: \"kubernetes.io/projected/94c593c7-be5a-4eef-94a2-5ea1091e93bd-kube-api-access-7dm6h\") pod \"maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:43.825098 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.825002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-maas-api-tls\") pod \"maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:43.825238 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:02:43.825114    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-serving-cert: secret "maas-api-e2e-migrate-d3676176-serving-cert" not found
Sep 11 04:02:43.825238 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:02:43.825164    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-maas-api-tls podName:94c593c7-be5a-4eef-94a2-5ea1091e93bd nodeName:}" failed. No retries permitted until 2026-09-11 04:02:44.3251493 +0000 UTC m=+756.947542306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-maas-api-tls") pod "maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m" (UID: "94c593c7-be5a-4eef-94a2-5ea1091e93bd") : secret "maas-api-e2e-migrate-d3676176-serving-cert" not found
Sep 11 04:02:43.825735 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.825709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c593c7-be5a-4eef-94a2-5ea1091e93bd-openshift-service-ca\") pod \"maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:43.827431 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.827406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-metrics-certs\") pod \"maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:43.835762 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.835739    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dm6h\" (UniqueName: \"kubernetes.io/projected/94c593c7-be5a-4eef-94a2-5ea1091e93bd-kube-api-access-7dm6h\") pod \"maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:43.840497 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.840471    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:43.969106 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:43.969000    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:44.200414 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:44.200375    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:44.301397 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:44.301295    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:44.330312 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:44.330280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-maas-api-tls\") pod \"maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:44.332897 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:44.332867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-maas-api-tls\") pod \"maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:44.620370 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:44.620279    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:44.894795 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:44.894762    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"]
Sep 11 04:02:44.896792 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:02:44.896764    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c593c7_be5a_4eef_94a2_5ea1091e93bd.slice/crio-df3790a5424f2bf232565dd96ab6821a513a6dd874dcb6a9bdbe69f0e10c1d70 WatchSource:0}: Error finding container df3790a5424f2bf232565dd96ab6821a513a6dd874dcb6a9bdbe69f0e10c1d70: Status 404 returned error can't find the container with id df3790a5424f2bf232565dd96ab6821a513a6dd874dcb6a9bdbe69f0e10c1d70
Sep 11 04:02:45.479900 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:45.479827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m" event={"ID":"94c593c7-be5a-4eef-94a2-5ea1091e93bd","Type":"ContainerStarted","Data":"3596943eb48f5dc8f838868e4983ecd1497ce2b84ad61ebab5ede0541f20448d"}
Sep 11 04:02:45.479900 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:45.479879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m" event={"ID":"94c593c7-be5a-4eef-94a2-5ea1091e93bd","Type":"ContainerStarted","Data":"df3790a5424f2bf232565dd96ab6821a513a6dd874dcb6a9bdbe69f0e10c1d70"}
Sep 11 04:02:45.480103 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:45.479916    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:02:45.500232 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:45.500185    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m" podStartSLOduration=2.182261494 podStartE2EDuration="2.5001685s" totalImagesPullingTime="317.907006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:02:45.498153597 +0000 UTC m=+758.120546643" watchObservedRunningTime="2026-09-11 04:02:45.5001685 +0000 UTC m=+758.122561528"
Sep 11 04:02:47.954737 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:47.954694    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:48.009077 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:48.009041    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:02:51.490125 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:02:51.490098    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:03:04.411860 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.411827    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"]
Sep 11 04:03:04.415285 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.415265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.418233 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.418211    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-92d75d22-kserve-self-signed-certs\""
Sep 11 04:03:04.422891 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.422866    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"]
Sep 11 04:03:04.436438 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.436410    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:04.533461 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.533431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-dshm\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.533461 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.533460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accb3763-bcec-4be4-a63d-5edb4c2fd127-tls-certs\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.533660 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.533491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-model-cache\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.533660 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.533565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-home\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.533660 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.533622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-tmp-dir\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.533660 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.533647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp772\" (UniqueName: \"kubernetes.io/projected/accb3763-bcec-4be4-a63d-5edb4c2fd127-kube-api-access-bp772\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.635000 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.634966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-dshm\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.635193 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.635010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accb3763-bcec-4be4-a63d-5edb4c2fd127-tls-certs\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.635193 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.635088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-model-cache\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.635193 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.635117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-home\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.635193 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.635172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-tmp-dir\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.635421 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.635202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp772\" (UniqueName: \"kubernetes.io/projected/accb3763-bcec-4be4-a63d-5edb4c2fd127-kube-api-access-bp772\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.635495 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.635471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-model-cache\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.635554 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.635534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-home\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.635689 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.635669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-tmp-dir\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.637202 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.637184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-dshm\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.637463 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.637447    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accb3763-bcec-4be4-a63d-5edb4c2fd127-tls-certs\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.642896 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.642879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp772\" (UniqueName: \"kubernetes.io/projected/accb3763-bcec-4be4-a63d-5edb4c2fd127-kube-api-access-bp772\") pod \"e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") " pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.726772 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.726705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:04.997480 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:04.997451    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"]
Sep 11 04:03:04.998913 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:03:04.998889    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaccb3763_bcec_4be4_a63d_5edb4c2fd127.slice/crio-accefdca3d6144a07f118940390dcee1d7e1bd4bae990ad2b08d95d03aa18b04 WatchSource:0}: Error finding container accefdca3d6144a07f118940390dcee1d7e1bd4bae990ad2b08d95d03aa18b04: Status 404 returned error can't find the container with id accefdca3d6144a07f118940390dcee1d7e1bd4bae990ad2b08d95d03aa18b04
Sep 11 04:03:05.588061 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:05.588009    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb" event={"ID":"accb3763-bcec-4be4-a63d-5edb4c2fd127","Type":"ContainerStarted","Data":"ab82632ee2237545887d0bc85f6d93a2afb57ce524af3116d9c89193ef6399fd"}
Sep 11 04:03:05.588438 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:05.588070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb" event={"ID":"accb3763-bcec-4be4-a63d-5edb4c2fd127","Type":"ContainerStarted","Data":"accefdca3d6144a07f118940390dcee1d7e1bd4bae990ad2b08d95d03aa18b04"}
Sep 11 04:03:05.605186 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:05.605142    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb" podStartSLOduration=1.605126802 podStartE2EDuration="1.605126802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:05.603797609 +0000 UTC m=+778.226190638" watchObservedRunningTime="2026-09-11 04:03:05.605126802 +0000 UTC m=+778.227519828"
Sep 11 04:03:14.210032 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.209979    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"]
Sep 11 04:03:14.210564 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.210258    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m" podUID="94c593c7-be5a-4eef-94a2-5ea1091e93bd" containerName="maas-api" containerID="cri-o://3596943eb48f5dc8f838868e4983ecd1497ce2b84ad61ebab5ede0541f20448d" gracePeriod=30
Sep 11 04:03:14.238081 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.238051    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:14.338530 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.338492    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:14.417209 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.417164    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:14.468702 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.468647    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:03:14.472774 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.472747    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:14.534079 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.534048    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dm6h\" (UniqueName: \"kubernetes.io/projected/94c593c7-be5a-4eef-94a2-5ea1091e93bd-kube-api-access-7dm6h\") pod \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") "
Sep 11 04:03:14.534257 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.534151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-metrics-certs\") pod \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") "
Sep 11 04:03:14.534257 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.534239    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c593c7-be5a-4eef-94a2-5ea1091e93bd-openshift-service-ca\") pod \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") "
Sep 11 04:03:14.534379 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.534312    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-maas-api-tls\") pod \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\" (UID: \"94c593c7-be5a-4eef-94a2-5ea1091e93bd\") "
Sep 11 04:03:14.534754 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.534705    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c593c7-be5a-4eef-94a2-5ea1091e93bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94c593c7-be5a-4eef-94a2-5ea1091e93bd" (UID: "94c593c7-be5a-4eef-94a2-5ea1091e93bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:03:14.538769 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.538736    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c593c7-be5a-4eef-94a2-5ea1091e93bd-kube-api-access-7dm6h" (OuterVolumeSpecName: "kube-api-access-7dm6h") pod "94c593c7-be5a-4eef-94a2-5ea1091e93bd" (UID: "94c593c7-be5a-4eef-94a2-5ea1091e93bd"). InnerVolumeSpecName "kube-api-access-7dm6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:03:14.539817 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.539777    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94c593c7-be5a-4eef-94a2-5ea1091e93bd" (UID: "94c593c7-be5a-4eef-94a2-5ea1091e93bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:03:14.539817 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.539798    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "94c593c7-be5a-4eef-94a2-5ea1091e93bd" (UID: "94c593c7-be5a-4eef-94a2-5ea1091e93bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:03:14.548855 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.548823    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:14.635166 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.635141    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c593c7-be5a-4eef-94a2-5ea1091e93bd-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:03:14.635166 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.635166    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:03:14.635368 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.635181    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dm6h\" (UniqueName: \"kubernetes.io/projected/94c593c7-be5a-4eef-94a2-5ea1091e93bd-kube-api-access-7dm6h\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:03:14.635368 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.635190    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94c593c7-be5a-4eef-94a2-5ea1091e93bd-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:03:14.635857 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.635832    2581 generic.go:356] "Generic (PLEG): container finished" podID="94c593c7-be5a-4eef-94a2-5ea1091e93bd" containerID="3596943eb48f5dc8f838868e4983ecd1497ce2b84ad61ebab5ede0541f20448d" exitCode=0
Sep 11 04:03:14.635933 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.635872    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"
Sep 11 04:03:14.635933 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.635875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m" event={"ID":"94c593c7-be5a-4eef-94a2-5ea1091e93bd","Type":"ContainerDied","Data":"3596943eb48f5dc8f838868e4983ecd1497ce2b84ad61ebab5ede0541f20448d"}
Sep 11 04:03:14.635933 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.635925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m" event={"ID":"94c593c7-be5a-4eef-94a2-5ea1091e93bd","Type":"ContainerDied","Data":"df3790a5424f2bf232565dd96ab6821a513a6dd874dcb6a9bdbe69f0e10c1d70"}
Sep 11 04:03:14.636075 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.635954    2581 scope.go:117] "RemoveContainer" containerID="3596943eb48f5dc8f838868e4983ecd1497ce2b84ad61ebab5ede0541f20448d"
Sep 11 04:03:14.644932 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.644916    2581 scope.go:117] "RemoveContainer" containerID="3596943eb48f5dc8f838868e4983ecd1497ce2b84ad61ebab5ede0541f20448d"
Sep 11 04:03:14.645218 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:03:14.645197    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3596943eb48f5dc8f838868e4983ecd1497ce2b84ad61ebab5ede0541f20448d\": container with ID starting with 3596943eb48f5dc8f838868e4983ecd1497ce2b84ad61ebab5ede0541f20448d not found: ID does not exist" containerID="3596943eb48f5dc8f838868e4983ecd1497ce2b84ad61ebab5ede0541f20448d"
Sep 11 04:03:14.645291 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.645225    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3596943eb48f5dc8f838868e4983ecd1497ce2b84ad61ebab5ede0541f20448d"} err="failed to get container status \"3596943eb48f5dc8f838868e4983ecd1497ce2b84ad61ebab5ede0541f20448d\": rpc error: code = NotFound desc = could not find container \"3596943eb48f5dc8f838868e4983ecd1497ce2b84ad61ebab5ede0541f20448d\": container with ID starting with 3596943eb48f5dc8f838868e4983ecd1497ce2b84ad61ebab5ede0541f20448d not found: ID does not exist"
Sep 11 04:03:14.666973 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.666949    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"]
Sep 11 04:03:14.672000 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.671978    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3676176-7b8777947f-b2v4m"]
Sep 11 04:03:14.727084 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.726973    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:14.727084 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.727066    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:14.739633 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.739613    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:14.958631 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:14.958592    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:15.242765 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:15.242736    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:15.569537 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:15.569504    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:15.653534 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:15.653506    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:03:15.862466 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:15.862377    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94c593c7-be5a-4eef-94a2-5ea1091e93bd" path="/var/lib/kubelet/pods/94c593c7-be5a-4eef-94a2-5ea1091e93bd/volumes"
Sep 11 04:03:18.118488 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:18.118438    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:18.218831 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:18.218798    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:19.408979 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.408947    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"]
Sep 11 04:03:19.409445 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.409430    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94c593c7-be5a-4eef-94a2-5ea1091e93bd" containerName="maas-api"
Sep 11 04:03:19.409492 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.409447    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="94c593c7-be5a-4eef-94a2-5ea1091e93bd" containerName="maas-api"
Sep 11 04:03:19.409536 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.409527    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="94c593c7-be5a-4eef-94a2-5ea1091e93bd" containerName="maas-api"
Sep 11 04:03:19.413799 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.413777    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:19.420298 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.420276    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1b0d05f3-serving-cert\""
Sep 11 04:03:19.425864 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.425837    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"]
Sep 11 04:03:19.461074 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.461043    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:19.483175 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.483148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-maas-api-tls\") pod \"maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:19.483280 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.483178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c5zz\" (UniqueName: \"kubernetes.io/projected/af4c2f8e-be5d-4dc3-a954-d47132b468b7-kube-api-access-8c5zz\") pod \"maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:19.483280 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.483211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4c2f8e-be5d-4dc3-a954-d47132b468b7-openshift-service-ca\") pod \"maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:19.483398 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.483304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-metrics-certs\") pod \"maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:19.561477 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.561446    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:19.584164 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.584133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-maas-api-tls\") pod \"maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:19.584295 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.584168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c5zz\" (UniqueName: \"kubernetes.io/projected/af4c2f8e-be5d-4dc3-a954-d47132b468b7-kube-api-access-8c5zz\") pod \"maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:19.584295 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.584219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4c2f8e-be5d-4dc3-a954-d47132b468b7-openshift-service-ca\") pod \"maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:19.584295 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.584273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-metrics-certs\") pod \"maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:19.584441 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:03:19.584274    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-serving-cert: secret "maas-api-e2e-ait-1b0d05f3-serving-cert" not found
Sep 11 04:03:19.584510 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:03:19.584497    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-maas-api-tls podName:af4c2f8e-be5d-4dc3-a954-d47132b468b7 nodeName:}" failed. No retries permitted until 2026-09-11 04:03:20.084473077 +0000 UTC m=+792.706866105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-maas-api-tls") pod "maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm" (UID: "af4c2f8e-be5d-4dc3-a954-d47132b468b7") : secret "maas-api-e2e-ait-1b0d05f3-serving-cert" not found
Sep 11 04:03:19.584858 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.584837    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4c2f8e-be5d-4dc3-a954-d47132b468b7-openshift-service-ca\") pod \"maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:19.586526 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.586507    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-metrics-certs\") pod \"maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:19.593407 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.593389    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c5zz\" (UniqueName: \"kubernetes.io/projected/af4c2f8e-be5d-4dc3-a954-d47132b468b7-kube-api-access-8c5zz\") pod \"maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:19.966096 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:19.966061    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:20.065462 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:20.065431    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:20.088644 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:20.088612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-maas-api-tls\") pod \"maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:20.091297 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:20.091273    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-maas-api-tls\") pod \"maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:20.326747 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:20.326711    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:20.602155 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:20.602127    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"]
Sep 11 04:03:20.604092 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:03:20.604065    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4c2f8e_be5d_4dc3_a954_d47132b468b7.slice/crio-6b5051f41ccf0a25d19cc40c0c064c2598e895052e43c40682ec44fc2b4d176b WatchSource:0}: Error finding container 6b5051f41ccf0a25d19cc40c0c064c2598e895052e43c40682ec44fc2b4d176b: Status 404 returned error can't find the container with id 6b5051f41ccf0a25d19cc40c0c064c2598e895052e43c40682ec44fc2b4d176b
Sep 11 04:03:20.669397 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:20.669366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm" event={"ID":"af4c2f8e-be5d-4dc3-a954-d47132b468b7","Type":"ContainerStarted","Data":"6b5051f41ccf0a25d19cc40c0c064c2598e895052e43c40682ec44fc2b4d176b"}
Sep 11 04:03:21.676207 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:21.676170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm" event={"ID":"af4c2f8e-be5d-4dc3-a954-d47132b468b7","Type":"ContainerStarted","Data":"285d173ba1b991242920fc7f2ba33578613244c40e6ecb1d582ccf3e67a07c51"}
Sep 11 04:03:21.676593 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:21.676301    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:21.693739 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:21.693694    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm" podStartSLOduration=2.27323788 podStartE2EDuration="2.693680933s" totalImagesPullingTime="420.443053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:21.692318807 +0000 UTC m=+794.314711835" watchObservedRunningTime="2026-09-11 04:03:21.693680933 +0000 UTC m=+794.316073959"
Sep 11 04:03:22.448169 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:22.448132    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:27.686052 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:27.686009    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:42.749146 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:42.749111    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:49.813528 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:49.813494    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"]
Sep 11 04:03:49.813925 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:49.813756    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm" podUID="af4c2f8e-be5d-4dc3-a954-d47132b468b7" containerName="maas-api" containerID="cri-o://285d173ba1b991242920fc7f2ba33578613244c40e6ecb1d582ccf3e67a07c51" gracePeriod=30
Sep 11 04:03:49.843403 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:49.843376    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:49.938921 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:49.938885    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:50.026524 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.026473    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:50.064371 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.064317    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:50.085253 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.085207    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c5zz\" (UniqueName: \"kubernetes.io/projected/af4c2f8e-be5d-4dc3-a954-d47132b468b7-kube-api-access-8c5zz\") pod \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") "
Sep 11 04:03:50.085425 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.085296    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-metrics-certs\") pod \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") "
Sep 11 04:03:50.085425 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.085368    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-maas-api-tls\") pod \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") "
Sep 11 04:03:50.085634 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.085471    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4c2f8e-be5d-4dc3-a954-d47132b468b7-openshift-service-ca\") pod \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\" (UID: \"af4c2f8e-be5d-4dc3-a954-d47132b468b7\") "
Sep 11 04:03:50.086079 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.085929    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:50.086249 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.086211    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4c2f8e-be5d-4dc3-a954-d47132b468b7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af4c2f8e-be5d-4dc3-a954-d47132b468b7" (UID: "af4c2f8e-be5d-4dc3-a954-d47132b468b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:03:50.088134 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.088098    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "af4c2f8e-be5d-4dc3-a954-d47132b468b7" (UID: "af4c2f8e-be5d-4dc3-a954-d47132b468b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:03:50.088271 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.088149    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4c2f8e-be5d-4dc3-a954-d47132b468b7-kube-api-access-8c5zz" (OuterVolumeSpecName: "kube-api-access-8c5zz") pod "af4c2f8e-be5d-4dc3-a954-d47132b468b7" (UID: "af4c2f8e-be5d-4dc3-a954-d47132b468b7"). InnerVolumeSpecName "kube-api-access-8c5zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:03:50.088930 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.088910    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af4c2f8e-be5d-4dc3-a954-d47132b468b7" (UID: "af4c2f8e-be5d-4dc3-a954-d47132b468b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:03:50.187145 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.187116    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4c2f8e-be5d-4dc3-a954-d47132b468b7-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:03:50.187145 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.187143    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c5zz\" (UniqueName: \"kubernetes.io/projected/af4c2f8e-be5d-4dc3-a954-d47132b468b7-kube-api-access-8c5zz\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:03:50.187338 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.187156    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:03:50.187338 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.187167    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4c2f8e-be5d-4dc3-a954-d47132b468b7-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:03:50.833445 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.833411    2581 generic.go:356] "Generic (PLEG): container finished" podID="af4c2f8e-be5d-4dc3-a954-d47132b468b7" containerID="285d173ba1b991242920fc7f2ba33578613244c40e6ecb1d582ccf3e67a07c51" exitCode=0
Sep 11 04:03:50.834044 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.833465    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm" event={"ID":"af4c2f8e-be5d-4dc3-a954-d47132b468b7","Type":"ContainerDied","Data":"285d173ba1b991242920fc7f2ba33578613244c40e6ecb1d582ccf3e67a07c51"}
Sep 11 04:03:50.834044 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.833489    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"
Sep 11 04:03:50.834044 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.833558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm" event={"ID":"af4c2f8e-be5d-4dc3-a954-d47132b468b7","Type":"ContainerDied","Data":"6b5051f41ccf0a25d19cc40c0c064c2598e895052e43c40682ec44fc2b4d176b"}
Sep 11 04:03:50.834044 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.833590    2581 scope.go:117] "RemoveContainer" containerID="285d173ba1b991242920fc7f2ba33578613244c40e6ecb1d582ccf3e67a07c51"
Sep 11 04:03:50.843765 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.843743    2581 scope.go:117] "RemoveContainer" containerID="285d173ba1b991242920fc7f2ba33578613244c40e6ecb1d582ccf3e67a07c51"
Sep 11 04:03:50.844336 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:03:50.844038    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285d173ba1b991242920fc7f2ba33578613244c40e6ecb1d582ccf3e67a07c51\": container with ID starting with 285d173ba1b991242920fc7f2ba33578613244c40e6ecb1d582ccf3e67a07c51 not found: ID does not exist" containerID="285d173ba1b991242920fc7f2ba33578613244c40e6ecb1d582ccf3e67a07c51"
Sep 11 04:03:50.844336 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.844075    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285d173ba1b991242920fc7f2ba33578613244c40e6ecb1d582ccf3e67a07c51"} err="failed to get container status \"285d173ba1b991242920fc7f2ba33578613244c40e6ecb1d582ccf3e67a07c51\": rpc error: code = NotFound desc = could not find container \"285d173ba1b991242920fc7f2ba33578613244c40e6ecb1d582ccf3e67a07c51\": container with ID starting with 285d173ba1b991242920fc7f2ba33578613244c40e6ecb1d582ccf3e67a07c51 not found: ID does not exist"
Sep 11 04:03:50.860870 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.860844    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"]
Sep 11 04:03:50.867846 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:50.867823    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b0d05f3-7bf5b4c85-7rlzm"]
Sep 11 04:03:51.137540 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:51.137418    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:51.521954 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:51.521854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:51.800067 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:51.799966    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:51.865496 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:51.865465    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af4c2f8e-be5d-4dc3-a954-d47132b468b7" path="/var/lib/kubelet/pods/af4c2f8e-be5d-4dc3-a954-d47132b468b7/volumes"
Sep 11 04:03:51.956145 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:51.956112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:54.187117 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.187073    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"]
Sep 11 04:03:54.187602 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.187563    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4c2f8e-be5d-4dc3-a954-d47132b468b7" containerName="maas-api"
Sep 11 04:03:54.187602 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.187578    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4c2f8e-be5d-4dc3-a954-d47132b468b7" containerName="maas-api"
Sep 11 04:03:54.187714 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.187672    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="af4c2f8e-be5d-4dc3-a954-d47132b468b7" containerName="maas-api"
Sep 11 04:03:54.191923 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.191904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.195797 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.195779    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-00a938\"/\"kube-root-ca.crt\""
Sep 11 04:03:54.195976 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.195919    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-00a938\"/\"default-dockercfg-jxd8v\""
Sep 11 04:03:54.196097 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.195984    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-00a938\"/\"test-model-00a938-kserve-self-signed-certs\""
Sep 11 04:03:54.196097 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.196014    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-00a938\"/\"openshift-service-ca.crt\""
Sep 11 04:03:54.201760 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.201740    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"]
Sep 11 04:03:54.210419 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.210394    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:54.225487 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.225467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-dshm\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.225588 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.225534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj5k9\" (UniqueName: \"kubernetes.io/projected/a11a8c3c-8f83-4283-b8c7-7b6a13991910-kube-api-access-tj5k9\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.225651 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.225613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-tmp-dir\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.225712 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.225656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-home\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.225712 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.225681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a11a8c3c-8f83-4283-b8c7-7b6a13991910-tls-certs\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.225798 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.225730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-model-cache\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.326266 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.326233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj5k9\" (UniqueName: \"kubernetes.io/projected/a11a8c3c-8f83-4283-b8c7-7b6a13991910-kube-api-access-tj5k9\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.326453 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.326315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-tmp-dir\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.326453 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.326355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-home\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.326453 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.326376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a11a8c3c-8f83-4283-b8c7-7b6a13991910-tls-certs\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.326453 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.326412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-model-cache\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.326678 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.326474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-dshm\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.326812 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.326789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-home\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.326870 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.326841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-model-cache\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.327051 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.327012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-tmp-dir\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.329355 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.329319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-dshm\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.329521 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.329489    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a11a8c3c-8f83-4283-b8c7-7b6a13991910-tls-certs\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.339518 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.339489    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj5k9\" (UniqueName: \"kubernetes.io/projected/a11a8c3c-8f83-4283-b8c7-7b6a13991910-kube-api-access-tj5k9\") pod \"test-model-00a938-kserve-5bf7cb7dcf-fvrck\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") " pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.379955 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.379923    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:54.501913 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.501831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:03:54.536497 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.536451    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:54.772336 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.772312    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"]
Sep 11 04:03:54.774880 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:03:54.774854    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda11a8c3c_8f83_4283_b8c7_7b6a13991910.slice/crio-7eadd7786e3406ded52a653d2dc4b3bef920dd6251ce640968dbf1930cef6b79 WatchSource:0}: Error finding container 7eadd7786e3406ded52a653d2dc4b3bef920dd6251ce640968dbf1930cef6b79: Status 404 returned error can't find the container with id 7eadd7786e3406ded52a653d2dc4b3bef920dd6251ce640968dbf1930cef6b79
Sep 11 04:03:54.856842 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.856783    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck" event={"ID":"a11a8c3c-8f83-4283-b8c7-7b6a13991910","Type":"ContainerStarted","Data":"9555acce202a157f67888141248c12085d1d8db9d8f51ab231b9084297254d8c"}
Sep 11 04:03:54.856842 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.856822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck" event={"ID":"a11a8c3c-8f83-4283-b8c7-7b6a13991910","Type":"ContainerStarted","Data":"7eadd7786e3406ded52a653d2dc4b3bef920dd6251ce640968dbf1930cef6b79"}
Sep 11 04:03:54.874214 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:54.874152    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck" podStartSLOduration=0.874136191 podStartE2EDuration="874.136191ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:54.873112886 +0000 UTC m=+827.495505914" watchObservedRunningTime="2026-09-11 04:03:54.874136191 +0000 UTC m=+827.496529218"
Sep 11 04:03:55.500241 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.500207    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"]
Sep 11 04:03:55.504278 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.504255    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:55.506750 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.506731    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0e17d64c-serving-cert\""
Sep 11 04:03:55.516945 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.516921    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"]
Sep 11 04:03:55.536236 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.536208    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:55.636656 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.636622    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:55.640728 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.640704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xf28\" (UniqueName: \"kubernetes.io/projected/e163de75-66d1-42d3-bc88-79ba36eb3d48-kube-api-access-9xf28\") pod \"maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:55.640838 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.640757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-metrics-certs\") pod \"maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:55.640886 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.640834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e163de75-66d1-42d3-bc88-79ba36eb3d48-openshift-service-ca\") pod \"maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:55.640886 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.640875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-maas-api-tls\") pod \"maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:55.742428 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.742396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xf28\" (UniqueName: \"kubernetes.io/projected/e163de75-66d1-42d3-bc88-79ba36eb3d48-kube-api-access-9xf28\") pod \"maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:55.742568 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.742447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-metrics-certs\") pod \"maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:55.742568 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.742493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e163de75-66d1-42d3-bc88-79ba36eb3d48-openshift-service-ca\") pod \"maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:55.742568 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.742538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-maas-api-tls\") pod \"maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:55.742736 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:03:55.742664    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-serving-cert: secret "maas-api-e2e-derive-0e17d64c-serving-cert" not found
Sep 11 04:03:55.742790 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:03:55.742744    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-maas-api-tls podName:e163de75-66d1-42d3-bc88-79ba36eb3d48 nodeName:}" failed. No retries permitted until 2026-09-11 04:03:56.242720144 +0000 UTC m=+828.865113166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-maas-api-tls") pod "maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw" (UID: "e163de75-66d1-42d3-bc88-79ba36eb3d48") : secret "maas-api-e2e-derive-0e17d64c-serving-cert" not found
Sep 11 04:03:55.743223 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.743201    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e163de75-66d1-42d3-bc88-79ba36eb3d48-openshift-service-ca\") pod \"maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:55.744934 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.744910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-metrics-certs\") pod \"maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:55.757944 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.757891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xf28\" (UniqueName: \"kubernetes.io/projected/e163de75-66d1-42d3-bc88-79ba36eb3d48-kube-api-access-9xf28\") pod \"maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:55.809057 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.808535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:55.985054 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:55.985009    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:56.086846 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:56.086817    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:56.247336 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:56.247303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-maas-api-tls\") pod \"maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:56.249735 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:56.249709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-maas-api-tls\") pod \"maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:56.416847 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:56.416769    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:56.689895 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:56.689870    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"]
Sep 11 04:03:56.691584 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:03:56.691556    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode163de75_66d1_42d3_bc88_79ba36eb3d48.slice/crio-6bce3e939fbeb8b43c5a0506b8a1d3433431a56d57b1a82b92c9e8de14a47565 WatchSource:0}: Error finding container 6bce3e939fbeb8b43c5a0506b8a1d3433431a56d57b1a82b92c9e8de14a47565: Status 404 returned error can't find the container with id 6bce3e939fbeb8b43c5a0506b8a1d3433431a56d57b1a82b92c9e8de14a47565
Sep 11 04:03:56.870417 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:56.870386    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw" event={"ID":"e163de75-66d1-42d3-bc88-79ba36eb3d48","Type":"ContainerStarted","Data":"6bce3e939fbeb8b43c5a0506b8a1d3433431a56d57b1a82b92c9e8de14a47565"}
Sep 11 04:03:57.779951 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:57.779921    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"]
Sep 11 04:03:57.795972 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:57.795937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:03:57.880585 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:57.880545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw" event={"ID":"e163de75-66d1-42d3-bc88-79ba36eb3d48","Type":"ContainerStarted","Data":"66d398599c5ca9ec8f1045fa894c3106017be9e945225652319b038ba75e5b5e"}
Sep 11 04:03:57.880756 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:57.880623    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:03:57.898640 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:57.898592    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw" podStartSLOduration=2.458462184 podStartE2EDuration="2.898578109s" totalImagesPullingTime="440.115925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:57.896988627 +0000 UTC m=+830.519381675" watchObservedRunningTime="2026-09-11 04:03:57.898578109 +0000 UTC m=+830.520971135"
Sep 11 04:03:58.881405 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:58.881370    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"]
Sep 11 04:03:58.903739 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:03:58.903713    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:03.891425 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:03.891394    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:04:04.502358 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:04.502320    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:04:04.502566 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:04.502379    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:04:04.514741 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:04.514717    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:04:04.928417 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:04.928380    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:04:06.595395 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:06.595366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:06.824590 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:06.824552    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:12.800341 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:12.800299    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv" podUID="727c386d-ed7f-4d43-aceb-32109e392716" containerName="main" containerID="cri-o://9ee5f8089268329e87ef1f01210544d4af482d192ae7bc8369674ee485b4fee4" gracePeriod=45
Sep 11 04:04:12.957362 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:12.957328    2581 generic.go:356] "Generic (PLEG): container finished" podID="727c386d-ed7f-4d43-aceb-32109e392716" containerID="9ee5f8089268329e87ef1f01210544d4af482d192ae7bc8369674ee485b4fee4" exitCode=0
Sep 11 04:04:12.957529 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:12.957367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv" event={"ID":"727c386d-ed7f-4d43-aceb-32109e392716","Type":"ContainerDied","Data":"9ee5f8089268329e87ef1f01210544d4af482d192ae7bc8369674ee485b4fee4"}
Sep 11 04:04:13.039853 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.039826    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:04:13.204982 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.204944    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727c386d-ed7f-4d43-aceb-32109e392716-tls-certs\") pod \"727c386d-ed7f-4d43-aceb-32109e392716\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") "
Sep 11 04:04:13.205189 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.205006    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-home\") pod \"727c386d-ed7f-4d43-aceb-32109e392716\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") "
Sep 11 04:04:13.205189 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.205059    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-model-cache\") pod \"727c386d-ed7f-4d43-aceb-32109e392716\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") "
Sep 11 04:04:13.205189 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.205082    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scr75\" (UniqueName: \"kubernetes.io/projected/727c386d-ed7f-4d43-aceb-32109e392716-kube-api-access-scr75\") pod \"727c386d-ed7f-4d43-aceb-32109e392716\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") "
Sep 11 04:04:13.205189 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.205145    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-tmp-dir\") pod \"727c386d-ed7f-4d43-aceb-32109e392716\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") "
Sep 11 04:04:13.205402 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.205205    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-dshm\") pod \"727c386d-ed7f-4d43-aceb-32109e392716\" (UID: \"727c386d-ed7f-4d43-aceb-32109e392716\") "
Sep 11 04:04:13.205402 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.205361    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-home" (OuterVolumeSpecName: "home") pod "727c386d-ed7f-4d43-aceb-32109e392716" (UID: "727c386d-ed7f-4d43-aceb-32109e392716"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:13.205402 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.205357    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-model-cache" (OuterVolumeSpecName: "model-cache") pod "727c386d-ed7f-4d43-aceb-32109e392716" (UID: "727c386d-ed7f-4d43-aceb-32109e392716"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:13.205523 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.205409    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "727c386d-ed7f-4d43-aceb-32109e392716" (UID: "727c386d-ed7f-4d43-aceb-32109e392716"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:13.205689 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.205661    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-home\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:13.205818 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.205690    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-model-cache\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:13.205818 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.205706    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-tmp-dir\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:13.207457 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.207427    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/727c386d-ed7f-4d43-aceb-32109e392716-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "727c386d-ed7f-4d43-aceb-32109e392716" (UID: "727c386d-ed7f-4d43-aceb-32109e392716"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:04:13.207580 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.207488    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-dshm" (OuterVolumeSpecName: "dshm") pod "727c386d-ed7f-4d43-aceb-32109e392716" (UID: "727c386d-ed7f-4d43-aceb-32109e392716"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:13.207647 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.207617    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727c386d-ed7f-4d43-aceb-32109e392716-kube-api-access-scr75" (OuterVolumeSpecName: "kube-api-access-scr75") pod "727c386d-ed7f-4d43-aceb-32109e392716" (UID: "727c386d-ed7f-4d43-aceb-32109e392716"). InnerVolumeSpecName "kube-api-access-scr75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:13.306421 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.306391    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727c386d-ed7f-4d43-aceb-32109e392716-tls-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:13.306421 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.306416    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scr75\" (UniqueName: \"kubernetes.io/projected/727c386d-ed7f-4d43-aceb-32109e392716-kube-api-access-scr75\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:13.306421 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.306427    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727c386d-ed7f-4d43-aceb-32109e392716-dshm\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:13.900732 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.900700    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb" podUID="accb3763-bcec-4be4-a63d-5edb4c2fd127" containerName="main" containerID="cri-o://ab82632ee2237545887d0bc85f6d93a2afb57ce524af3116d9c89193ef6399fd" gracePeriod=45
Sep 11 04:04:13.964131 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.964100    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"
Sep 11 04:04:13.964131 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.964119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv" event={"ID":"727c386d-ed7f-4d43-aceb-32109e392716","Type":"ContainerDied","Data":"6088b982d96eec235addf1a16926814aff95d830ec4a6e340ba0c5456ca727be"}
Sep 11 04:04:13.964289 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.964171    2581 scope.go:117] "RemoveContainer" containerID="9ee5f8089268329e87ef1f01210544d4af482d192ae7bc8369674ee485b4fee4"
Sep 11 04:04:13.985787 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.985762    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"]
Sep 11 04:04:13.989512 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:13.989490    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-92d75d22-kserve-857845995b-5twpv"]
Sep 11 04:04:14.150567 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.150539    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:04:14.316964 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.316936    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-tmp-dir\") pod \"accb3763-bcec-4be4-a63d-5edb4c2fd127\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") "
Sep 11 04:04:14.317146 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.316974    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp772\" (UniqueName: \"kubernetes.io/projected/accb3763-bcec-4be4-a63d-5edb4c2fd127-kube-api-access-bp772\") pod \"accb3763-bcec-4be4-a63d-5edb4c2fd127\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") "
Sep 11 04:04:14.317146 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.316993    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-home\") pod \"accb3763-bcec-4be4-a63d-5edb4c2fd127\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") "
Sep 11 04:04:14.317146 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.317069    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-dshm\") pod \"accb3763-bcec-4be4-a63d-5edb4c2fd127\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") "
Sep 11 04:04:14.317146 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.317134    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-model-cache\") pod \"accb3763-bcec-4be4-a63d-5edb4c2fd127\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") "
Sep 11 04:04:14.317360 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.317174    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accb3763-bcec-4be4-a63d-5edb4c2fd127-tls-certs\") pod \"accb3763-bcec-4be4-a63d-5edb4c2fd127\" (UID: \"accb3763-bcec-4be4-a63d-5edb4c2fd127\") "
Sep 11 04:04:14.317360 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.317206    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "accb3763-bcec-4be4-a63d-5edb4c2fd127" (UID: "accb3763-bcec-4be4-a63d-5edb4c2fd127"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:14.317360 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.317314    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-home" (OuterVolumeSpecName: "home") pod "accb3763-bcec-4be4-a63d-5edb4c2fd127" (UID: "accb3763-bcec-4be4-a63d-5edb4c2fd127"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:14.317520 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.317419    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-model-cache" (OuterVolumeSpecName: "model-cache") pod "accb3763-bcec-4be4-a63d-5edb4c2fd127" (UID: "accb3763-bcec-4be4-a63d-5edb4c2fd127"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:14.317574 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.317531    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-model-cache\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:14.317574 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.317552    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-tmp-dir\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:14.317574 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.317566    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-home\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:14.319158 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.319129    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-dshm" (OuterVolumeSpecName: "dshm") pod "accb3763-bcec-4be4-a63d-5edb4c2fd127" (UID: "accb3763-bcec-4be4-a63d-5edb4c2fd127"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:14.319287 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.319190    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/accb3763-bcec-4be4-a63d-5edb4c2fd127-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "accb3763-bcec-4be4-a63d-5edb4c2fd127" (UID: "accb3763-bcec-4be4-a63d-5edb4c2fd127"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:04:14.319287 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.319210    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/accb3763-bcec-4be4-a63d-5edb4c2fd127-kube-api-access-bp772" (OuterVolumeSpecName: "kube-api-access-bp772") pod "accb3763-bcec-4be4-a63d-5edb4c2fd127" (UID: "accb3763-bcec-4be4-a63d-5edb4c2fd127"). InnerVolumeSpecName "kube-api-access-bp772". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:14.418259 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.418194    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp772\" (UniqueName: \"kubernetes.io/projected/accb3763-bcec-4be4-a63d-5edb4c2fd127-kube-api-access-bp772\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:14.418259 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.418218    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accb3763-bcec-4be4-a63d-5edb4c2fd127-dshm\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:14.418259 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.418227    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accb3763-bcec-4be4-a63d-5edb4c2fd127-tls-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:14.971647 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.971609    2581 generic.go:356] "Generic (PLEG): container finished" podID="accb3763-bcec-4be4-a63d-5edb4c2fd127" containerID="ab82632ee2237545887d0bc85f6d93a2afb57ce524af3116d9c89193ef6399fd" exitCode=0
Sep 11 04:04:14.971647 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.971648    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"
Sep 11 04:04:14.972122 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.971660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb" event={"ID":"accb3763-bcec-4be4-a63d-5edb4c2fd127","Type":"ContainerDied","Data":"ab82632ee2237545887d0bc85f6d93a2afb57ce524af3116d9c89193ef6399fd"}
Sep 11 04:04:14.972122 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.971690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb" event={"ID":"accb3763-bcec-4be4-a63d-5edb4c2fd127","Type":"ContainerDied","Data":"accefdca3d6144a07f118940390dcee1d7e1bd4bae990ad2b08d95d03aa18b04"}
Sep 11 04:04:14.972122 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.971708    2581 scope.go:117] "RemoveContainer" containerID="ab82632ee2237545887d0bc85f6d93a2afb57ce524af3116d9c89193ef6399fd"
Sep 11 04:04:14.981538 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.981516    2581 scope.go:117] "RemoveContainer" containerID="ab82632ee2237545887d0bc85f6d93a2afb57ce524af3116d9c89193ef6399fd"
Sep 11 04:04:14.981794 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:04:14.981774    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab82632ee2237545887d0bc85f6d93a2afb57ce524af3116d9c89193ef6399fd\": container with ID starting with ab82632ee2237545887d0bc85f6d93a2afb57ce524af3116d9c89193ef6399fd not found: ID does not exist" containerID="ab82632ee2237545887d0bc85f6d93a2afb57ce524af3116d9c89193ef6399fd"
Sep 11 04:04:14.981835 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.981802    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab82632ee2237545887d0bc85f6d93a2afb57ce524af3116d9c89193ef6399fd"} err="failed to get container status \"ab82632ee2237545887d0bc85f6d93a2afb57ce524af3116d9c89193ef6399fd\": rpc error: code = NotFound desc = could not find container \"ab82632ee2237545887d0bc85f6d93a2afb57ce524af3116d9c89193ef6399fd\": container with ID starting with ab82632ee2237545887d0bc85f6d93a2afb57ce524af3116d9c89193ef6399fd not found: ID does not exist"
Sep 11 04:04:14.994311 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.994288    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"]
Sep 11 04:04:14.996205 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:14.996184    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-92d75d22-kserve-bb4b8c54b-vqtwb"]
Sep 11 04:04:15.862355 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:15.862318    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="727c386d-ed7f-4d43-aceb-32109e392716" path="/var/lib/kubelet/pods/727c386d-ed7f-4d43-aceb-32109e392716/volumes"
Sep 11 04:04:15.862727 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:15.862713    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="accb3763-bcec-4be4-a63d-5edb4c2fd127" path="/var/lib/kubelet/pods/accb3763-bcec-4be4-a63d-5edb4c2fd127/volumes"
Sep 11 04:04:16.366721 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:16.366685    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:17.050510 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:17.050455    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:17.150006 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:17.149969    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:17.253329 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:17.253291    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:17.432410 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:17.432381    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:17.501852 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:17.501818    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:17.763506 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:17.763428    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:24.146822 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:24.146780    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:24.248338 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:24.248303    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:24.385285 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:24.385235    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:24.850377 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:24.850339    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:25.116810 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:25.116723    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:25.492498 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:25.492409    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:26.353549 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.353515    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"]
Sep 11 04:04:26.353788 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.353760    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw" podUID="e163de75-66d1-42d3-bc88-79ba36eb3d48" containerName="maas-api" containerID="cri-o://66d398599c5ca9ec8f1045fa894c3106017be9e945225652319b038ba75e5b5e" gracePeriod=30
Sep 11 04:04:26.382378 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.382340    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:26.498402 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.498367    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:26.603621 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.603566    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:04:26.622166 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.622139    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:26.637838 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.637814    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xf28\" (UniqueName: \"kubernetes.io/projected/e163de75-66d1-42d3-bc88-79ba36eb3d48-kube-api-access-9xf28\") pod \"e163de75-66d1-42d3-bc88-79ba36eb3d48\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") "
Sep 11 04:04:26.637949 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.637851    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-metrics-certs\") pod \"e163de75-66d1-42d3-bc88-79ba36eb3d48\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") "
Sep 11 04:04:26.638064 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.638046    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e163de75-66d1-42d3-bc88-79ba36eb3d48-openshift-service-ca\") pod \"e163de75-66d1-42d3-bc88-79ba36eb3d48\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") "
Sep 11 04:04:26.638118 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.638086    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-maas-api-tls\") pod \"e163de75-66d1-42d3-bc88-79ba36eb3d48\" (UID: \"e163de75-66d1-42d3-bc88-79ba36eb3d48\") "
Sep 11 04:04:26.638472 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.638442    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e163de75-66d1-42d3-bc88-79ba36eb3d48-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e163de75-66d1-42d3-bc88-79ba36eb3d48" (UID: "e163de75-66d1-42d3-bc88-79ba36eb3d48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:26.639904 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.639874    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e163de75-66d1-42d3-bc88-79ba36eb3d48" (UID: "e163de75-66d1-42d3-bc88-79ba36eb3d48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:04:26.640012 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.639944    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e163de75-66d1-42d3-bc88-79ba36eb3d48-kube-api-access-9xf28" (OuterVolumeSpecName: "kube-api-access-9xf28") pod "e163de75-66d1-42d3-bc88-79ba36eb3d48" (UID: "e163de75-66d1-42d3-bc88-79ba36eb3d48"). InnerVolumeSpecName "kube-api-access-9xf28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:26.640111 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.640043    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e163de75-66d1-42d3-bc88-79ba36eb3d48" (UID: "e163de75-66d1-42d3-bc88-79ba36eb3d48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:04:26.699299 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.699267    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:26.739450 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.739420    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xf28\" (UniqueName: \"kubernetes.io/projected/e163de75-66d1-42d3-bc88-79ba36eb3d48-kube-api-access-9xf28\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:26.739450 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.739444    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:26.739450 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.739454    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e163de75-66d1-42d3-bc88-79ba36eb3d48-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:26.739661 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.739464    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e163de75-66d1-42d3-bc88-79ba36eb3d48-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:04:26.808573 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:26.808530    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:27.040895 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:27.040800    2581 generic.go:356] "Generic (PLEG): container finished" podID="e163de75-66d1-42d3-bc88-79ba36eb3d48" containerID="66d398599c5ca9ec8f1045fa894c3106017be9e945225652319b038ba75e5b5e" exitCode=0
Sep 11 04:04:27.040895 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:27.040841    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"
Sep 11 04:04:27.040895 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:27.040883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw" event={"ID":"e163de75-66d1-42d3-bc88-79ba36eb3d48","Type":"ContainerDied","Data":"66d398599c5ca9ec8f1045fa894c3106017be9e945225652319b038ba75e5b5e"}
Sep 11 04:04:27.041178 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:27.040935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw" event={"ID":"e163de75-66d1-42d3-bc88-79ba36eb3d48","Type":"ContainerDied","Data":"6bce3e939fbeb8b43c5a0506b8a1d3433431a56d57b1a82b92c9e8de14a47565"}
Sep 11 04:04:27.041178 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:27.040958    2581 scope.go:117] "RemoveContainer" containerID="66d398599c5ca9ec8f1045fa894c3106017be9e945225652319b038ba75e5b5e"
Sep 11 04:04:27.050152 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:27.050137    2581 scope.go:117] "RemoveContainer" containerID="66d398599c5ca9ec8f1045fa894c3106017be9e945225652319b038ba75e5b5e"
Sep 11 04:04:27.050432 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:04:27.050411    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d398599c5ca9ec8f1045fa894c3106017be9e945225652319b038ba75e5b5e\": container with ID starting with 66d398599c5ca9ec8f1045fa894c3106017be9e945225652319b038ba75e5b5e not found: ID does not exist" containerID="66d398599c5ca9ec8f1045fa894c3106017be9e945225652319b038ba75e5b5e"
Sep 11 04:04:27.050512 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:27.050439    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d398599c5ca9ec8f1045fa894c3106017be9e945225652319b038ba75e5b5e"} err="failed to get container status \"66d398599c5ca9ec8f1045fa894c3106017be9e945225652319b038ba75e5b5e\": rpc error: code = NotFound desc = could not find container \"66d398599c5ca9ec8f1045fa894c3106017be9e945225652319b038ba75e5b5e\": container with ID starting with 66d398599c5ca9ec8f1045fa894c3106017be9e945225652319b038ba75e5b5e not found: ID does not exist"
Sep 11 04:04:27.062460 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:27.062438    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"]
Sep 11 04:04:27.068500 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:27.068480    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0e17d64c-59d549bf95-7hvlw"]
Sep 11 04:04:27.439088 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:27.439051    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:27.532408 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:27.532375    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:27.635463 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:27.635427    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:27.862548 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:27.862517    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e163de75-66d1-42d3-bc88-79ba36eb3d48" path="/var/lib/kubelet/pods/e163de75-66d1-42d3-bc88-79ba36eb3d48/volumes"
Sep 11 04:04:28.589405 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:28.589371    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:28.688887 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:28.688847    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:28.969923 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:28.969830    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:29.705628 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.705591    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"]
Sep 11 04:04:29.706190 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.706171    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="accb3763-bcec-4be4-a63d-5edb4c2fd127" containerName="main"
Sep 11 04:04:29.706190 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.706191    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="accb3763-bcec-4be4-a63d-5edb4c2fd127" containerName="main"
Sep 11 04:04:29.706314 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.706212    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727c386d-ed7f-4d43-aceb-32109e392716" containerName="main"
Sep 11 04:04:29.706314 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.706217    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="727c386d-ed7f-4d43-aceb-32109e392716" containerName="main"
Sep 11 04:04:29.706314 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.706224    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e163de75-66d1-42d3-bc88-79ba36eb3d48" containerName="maas-api"
Sep 11 04:04:29.706314 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.706230    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e163de75-66d1-42d3-bc88-79ba36eb3d48" containerName="maas-api"
Sep 11 04:04:29.706314 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.706294    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="727c386d-ed7f-4d43-aceb-32109e392716" containerName="main"
Sep 11 04:04:29.706314 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.706304    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="accb3763-bcec-4be4-a63d-5edb4c2fd127" containerName="main"
Sep 11 04:04:29.706314 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.706311    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e163de75-66d1-42d3-bc88-79ba36eb3d48" containerName="maas-api"
Sep 11 04:04:29.711063 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.711042    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:29.713527 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.713507    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-557c41-serving-cert\""
Sep 11 04:04:29.722689 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.722666    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"]
Sep 11 04:04:29.739949 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.739903    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:29.768149 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.768117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-maas-api-tls\") pod \"maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:29.768270 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.768166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmfcv\" (UniqueName: \"kubernetes.io/projected/d671ef27-e518-4ca3-abf3-f899e4915ab0-kube-api-access-cmfcv\") pod \"maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:29.768270 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.768193    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d671ef27-e518-4ca3-abf3-f899e4915ab0-openshift-service-ca\") pod \"maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:29.768383 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.768291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-metrics-certs\") pod \"maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:29.857923 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.857885    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:29.869126 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.869096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-maas-api-tls\") pod \"maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:29.869287 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.869148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmfcv\" (UniqueName: \"kubernetes.io/projected/d671ef27-e518-4ca3-abf3-f899e4915ab0-kube-api-access-cmfcv\") pod \"maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:29.869287 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.869252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d671ef27-e518-4ca3-abf3-f899e4915ab0-openshift-service-ca\") pod \"maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:29.869287 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:04:29.869277    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-serving-cert: secret "maas-api-e2e-explicit-557c41-serving-cert" not found
Sep 11 04:04:29.869431 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:04:29.869351    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-maas-api-tls podName:d671ef27-e518-4ca3-abf3-f899e4915ab0 nodeName:}" failed. No retries permitted until 2026-09-11 04:04:30.369328412 +0000 UTC m=+862.991721435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-maas-api-tls") pod "maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr" (UID: "d671ef27-e518-4ca3-abf3-f899e4915ab0") : secret "maas-api-e2e-explicit-557c41-serving-cert" not found
Sep 11 04:04:29.869431 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.869399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-metrics-certs\") pod \"maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:29.869870 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.869853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d671ef27-e518-4ca3-abf3-f899e4915ab0-openshift-service-ca\") pod \"maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:29.871734 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.871716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-metrics-certs\") pod \"maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:29.877330 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.877311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmfcv\" (UniqueName: \"kubernetes.io/projected/d671ef27-e518-4ca3-abf3-f899e4915ab0-kube-api-access-cmfcv\") pod \"maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:30.000083 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:29.999964    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:30.303280 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:30.303248    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:30.373908 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:30.373877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-maas-api-tls\") pod \"maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:30.376193 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:30.376170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-maas-api-tls\") pod \"maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:30.406937 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:30.406904    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:30.622851 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:30.622765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:30.896190 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:30.896164    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"]
Sep 11 04:04:30.898073 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:04:30.898047    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd671ef27_e518_4ca3_abf3_f899e4915ab0.slice/crio-1e9e31c8a4e30512ef3199657d489f47d9a3d2735855edfdc67980c999829c15 WatchSource:0}: Error finding container 1e9e31c8a4e30512ef3199657d489f47d9a3d2735855edfdc67980c999829c15: Status 404 returned error can't find the container with id 1e9e31c8a4e30512ef3199657d489f47d9a3d2735855edfdc67980c999829c15
Sep 11 04:04:31.064591 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:31.064556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr" event={"ID":"d671ef27-e518-4ca3-abf3-f899e4915ab0","Type":"ContainerStarted","Data":"1e9e31c8a4e30512ef3199657d489f47d9a3d2735855edfdc67980c999829c15"}
Sep 11 04:04:32.072763 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:32.072723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr" event={"ID":"d671ef27-e518-4ca3-abf3-f899e4915ab0","Type":"ContainerStarted","Data":"8503ed8a747f41f3a82e012409626aa207d680f1105abc622e97e4eba487ab16"}
Sep 11 04:04:32.073159 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:32.072840    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:32.090329 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:32.090277    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr" podStartSLOduration=2.582350607 podStartE2EDuration="3.090263968s" totalImagesPullingTime="507.913361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:04:32.088084925 +0000 UTC m=+864.710477970" watchObservedRunningTime="2026-09-11 04:04:32.090263968 +0000 UTC m=+864.712656995"
Sep 11 04:04:34.636535 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:34.636490    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:34.757627 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:34.757587    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:34.850641 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:34.850610    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:34.948147 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:34.948064    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:35.286588 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:35.286472    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:35.626143 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:35.626106    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:38.083114 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:38.083083    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:04:46.232856 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:46.232820    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:50.718669 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:50.718625    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:04:53.405806 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:04:53.405760    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:01.195574 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:01.195489    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:06.954773 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:06.954728    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:08.699144 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:08.699086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:20.818872 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:20.818829    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:22.901699 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:22.901638    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:25.334933 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:25.334890    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:28.813439 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:28.813398    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:28.914125 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:28.914089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:29.022513 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:29.022470    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:36.012108 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.012065    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"]
Sep 11 04:05:36.012628 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.012316    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr" podUID="d671ef27-e518-4ca3-abf3-f899e4915ab0" containerName="maas-api" containerID="cri-o://8503ed8a747f41f3a82e012409626aa207d680f1105abc622e97e4eba487ab16" gracePeriod=30
Sep 11 04:05:36.043428 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.043400    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:36.111860 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.111824    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:36.212178 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.212140    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:36.253359 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.253336    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:05:36.312982 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.312950    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:36.409320 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.409287    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-maas-api-tls\") pod \"d671ef27-e518-4ca3-abf3-f899e4915ab0\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") "
Sep 11 04:05:36.409502 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.409343    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-metrics-certs\") pod \"d671ef27-e518-4ca3-abf3-f899e4915ab0\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") "
Sep 11 04:05:36.409502 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.409370    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d671ef27-e518-4ca3-abf3-f899e4915ab0-openshift-service-ca\") pod \"d671ef27-e518-4ca3-abf3-f899e4915ab0\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") "
Sep 11 04:05:36.409502 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.409443    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmfcv\" (UniqueName: \"kubernetes.io/projected/d671ef27-e518-4ca3-abf3-f899e4915ab0-kube-api-access-cmfcv\") pod \"d671ef27-e518-4ca3-abf3-f899e4915ab0\" (UID: \"d671ef27-e518-4ca3-abf3-f899e4915ab0\") "
Sep 11 04:05:36.409798 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.409764    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d671ef27-e518-4ca3-abf3-f899e4915ab0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d671ef27-e518-4ca3-abf3-f899e4915ab0" (UID: "d671ef27-e518-4ca3-abf3-f899e4915ab0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:05:36.411474 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.411444    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d671ef27-e518-4ca3-abf3-f899e4915ab0" (UID: "d671ef27-e518-4ca3-abf3-f899e4915ab0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:05:36.411474 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.411456    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d671ef27-e518-4ca3-abf3-f899e4915ab0-kube-api-access-cmfcv" (OuterVolumeSpecName: "kube-api-access-cmfcv") pod "d671ef27-e518-4ca3-abf3-f899e4915ab0" (UID: "d671ef27-e518-4ca3-abf3-f899e4915ab0"). InnerVolumeSpecName "kube-api-access-cmfcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:05:36.411474 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.411463    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d671ef27-e518-4ca3-abf3-f899e4915ab0" (UID: "d671ef27-e518-4ca3-abf3-f899e4915ab0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:05:36.419126 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.419100    2581 generic.go:356] "Generic (PLEG): container finished" podID="d671ef27-e518-4ca3-abf3-f899e4915ab0" containerID="8503ed8a747f41f3a82e012409626aa207d680f1105abc622e97e4eba487ab16" exitCode=0
Sep 11 04:05:36.419233 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.419181    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"
Sep 11 04:05:36.419233 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.419195    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr" event={"ID":"d671ef27-e518-4ca3-abf3-f899e4915ab0","Type":"ContainerDied","Data":"8503ed8a747f41f3a82e012409626aa207d680f1105abc622e97e4eba487ab16"}
Sep 11 04:05:36.419326 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.419261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr" event={"ID":"d671ef27-e518-4ca3-abf3-f899e4915ab0","Type":"ContainerDied","Data":"1e9e31c8a4e30512ef3199657d489f47d9a3d2735855edfdc67980c999829c15"}
Sep 11 04:05:36.419326 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.419293    2581 scope.go:117] "RemoveContainer" containerID="8503ed8a747f41f3a82e012409626aa207d680f1105abc622e97e4eba487ab16"
Sep 11 04:05:36.439213 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.439195    2581 scope.go:117] "RemoveContainer" containerID="8503ed8a747f41f3a82e012409626aa207d680f1105abc622e97e4eba487ab16"
Sep 11 04:05:36.439471 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:05:36.439452    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8503ed8a747f41f3a82e012409626aa207d680f1105abc622e97e4eba487ab16\": container with ID starting with 8503ed8a747f41f3a82e012409626aa207d680f1105abc622e97e4eba487ab16 not found: ID does not exist" containerID="8503ed8a747f41f3a82e012409626aa207d680f1105abc622e97e4eba487ab16"
Sep 11 04:05:36.439530 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.439479    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8503ed8a747f41f3a82e012409626aa207d680f1105abc622e97e4eba487ab16"} err="failed to get container status \"8503ed8a747f41f3a82e012409626aa207d680f1105abc622e97e4eba487ab16\": rpc error: code = NotFound desc = could not find container \"8503ed8a747f41f3a82e012409626aa207d680f1105abc622e97e4eba487ab16\": container with ID starting with 8503ed8a747f41f3a82e012409626aa207d680f1105abc622e97e4eba487ab16 not found: ID does not exist"
Sep 11 04:05:36.452922 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.452898    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"]
Sep 11 04:05:36.457897 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.457876    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-557c41-6ff46d99cc-6wqxr"]
Sep 11 04:05:36.510562 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.510536    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmfcv\" (UniqueName: \"kubernetes.io/projected/d671ef27-e518-4ca3-abf3-f899e4915ab0-kube-api-access-cmfcv\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:05:36.510562 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.510558    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:05:36.510696 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.510573    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d671ef27-e518-4ca3-abf3-f899e4915ab0-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:05:36.510696 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.510585    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d671ef27-e518-4ca3-abf3-f899e4915ab0-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:05:36.574473 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:36.574398    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:37.862003 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:37.861973    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d671ef27-e518-4ca3-abf3-f899e4915ab0" path="/var/lib/kubelet/pods/d671ef27-e518-4ca3-abf3-f899e4915ab0/volumes"
Sep 11 04:05:39.476457 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:39.476419    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:40.199427 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:40.199393    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:42.779540 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:42.779491    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:42.881005 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:42.880972    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:43.037377 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:43.037284    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:43.436783 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:43.436746    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:43.734841 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:43.734740    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:44.037015 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:44.036932    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:45.029186 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:45.029156    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:45.169944 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:45.169906    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:45.285376 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:45.285300    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:46.710287 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:46.710253    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:46.801855 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:46.801819    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:46.901982 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:46.901942    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:47.484629 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:47.484594    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:47.580306 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:47.580273    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:47.680182 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:47.680143    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:49.020444 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:49.020409    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:54.635416 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:54.635370    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:54.694660 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:54.694624    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:55.288311 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:55.288275    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:59.395682 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.395645    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"]
Sep 11 04:05:59.396143 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.396098    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d671ef27-e518-4ca3-abf3-f899e4915ab0" containerName="maas-api"
Sep 11 04:05:59.396143 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.396112    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d671ef27-e518-4ca3-abf3-f899e4915ab0" containerName="maas-api"
Sep 11 04:05:59.396263 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.396190    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d671ef27-e518-4ca3-abf3-f899e4915ab0" containerName="maas-api"
Sep 11 04:05:59.400359 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.400338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.403370 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.403329    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ce78b8\"/\"kube-root-ca.crt\""
Sep 11 04:05:59.404466 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.404450    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ce78b8\"/\"openshift-service-ca.crt\""
Sep 11 04:05:59.404843 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.404821    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ce78b8\"/\"default-dockercfg-b5qbx\""
Sep 11 04:05:59.404986 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.404889    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ce78b8\"/\"noten-model-ce78b8-kserve-self-signed-certs\""
Sep 11 04:05:59.409992 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.409969    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"]
Sep 11 04:05:59.418666 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.418638    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:05:59.533147 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.533105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-home\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.533312 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.533225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fb8x\" (UniqueName: \"kubernetes.io/projected/b546c384-3fa8-4b04-9e43-c099f9383930-kube-api-access-6fb8x\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.533312 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.533280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-model-cache\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.533389 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.533343    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-tmp-dir\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.533472 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.533448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b546c384-3fa8-4b04-9e43-c099f9383930-tls-certs\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.533508 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.533485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-dshm\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.634277 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.634237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-model-cache\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.634451 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.634288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-tmp-dir\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.634451 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.634335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b546c384-3fa8-4b04-9e43-c099f9383930-tls-certs\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.634451 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.634350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-dshm\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.634451 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.634378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-home\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.634451 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.634442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fb8x\" (UniqueName: \"kubernetes.io/projected/b546c384-3fa8-4b04-9e43-c099f9383930-kube-api-access-6fb8x\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.634722 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.634691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-model-cache\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.634781 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.634737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-tmp-dir\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.634843 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.634809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-home\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.636615 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.636594    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-dshm\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.636912 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.636891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b546c384-3fa8-4b04-9e43-c099f9383930-tls-certs\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.643417 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.643397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fb8x\" (UniqueName: \"kubernetes.io/projected/b546c384-3fa8-4b04-9e43-c099f9383930-kube-api-access-6fb8x\") pod \"noten-model-ce78b8-kserve-769986bcdc-g2gbd\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") " pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.711110 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.711048    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:05:59.982973 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:05:59.982947    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"]
Sep 11 04:05:59.984848 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:05:59.984818    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb546c384_3fa8_4b04_9e43_c099f9383930.slice/crio-e6e3fd1ef97ead9b25608dbbd9487726b9d5911e2f3adccf80598722db7e2b81 WatchSource:0}: Error finding container e6e3fd1ef97ead9b25608dbbd9487726b9d5911e2f3adccf80598722db7e2b81: Status 404 returned error can't find the container with id e6e3fd1ef97ead9b25608dbbd9487726b9d5911e2f3adccf80598722db7e2b81
Sep 11 04:06:00.553440 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:00.553404    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd" event={"ID":"b546c384-3fa8-4b04-9e43-c099f9383930","Type":"ContainerStarted","Data":"105d871a562d62907b25148b8886a2a4facbaf0f6b3a94ab88d09b95aae015dc"}
Sep 11 04:06:00.553440 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:00.553447    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd" event={"ID":"b546c384-3fa8-4b04-9e43-c099f9383930","Type":"ContainerStarted","Data":"e6e3fd1ef97ead9b25608dbbd9487726b9d5911e2f3adccf80598722db7e2b81"}
Sep 11 04:06:00.580052 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:00.579979    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd" podStartSLOduration=1.579965351 podStartE2EDuration="1.579965351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:06:00.579063533 +0000 UTC m=+953.201456558" watchObservedRunningTime="2026-09-11 04:06:00.579965351 +0000 UTC m=+953.202358378"
Sep 11 04:06:01.379913 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:01.379874    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:01.481514 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:01.481478    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:01.601148 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:01.601111    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:01.887198 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:01.887165    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:01.988208 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:01.988179    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:04.971796 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:04.971739    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:05.066355 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:05.066322    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:05.760807 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:05.760766    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:05.816768 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:05.816740    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:06.210603 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.210573    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"]
Sep 11 04:06:06.214274 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.214250    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:06.217735 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.217709    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2d69ee02-serving-cert\""
Sep 11 04:06:06.226859 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.226835    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"]
Sep 11 04:06:06.253872 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.253837    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:06.302146 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.302115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-maas-api-tls\") pod \"maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:06.302252 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.302149    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-openshift-service-ca\") pod \"maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:06.302329 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.302272    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khfwx\" (UniqueName: \"kubernetes.io/projected/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-kube-api-access-khfwx\") pod \"maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:06.302329 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.302311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-metrics-certs\") pod \"maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:06.379310 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.379281    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:06.403770 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.403743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-maas-api-tls\") pod \"maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:06.403887 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.403772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-openshift-service-ca\") pod \"maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:06.403887 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.403820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khfwx\" (UniqueName: \"kubernetes.io/projected/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-kube-api-access-khfwx\") pod \"maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:06.403887 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.403844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-metrics-certs\") pod \"maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:06.404545 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.404522    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-openshift-service-ca\") pod \"maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:06.406198 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.406175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-maas-api-tls\") pod \"maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:06.406384 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.406364    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-metrics-certs\") pod \"maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:06.421461 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.421440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khfwx\" (UniqueName: \"kubernetes.io/projected/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-kube-api-access-khfwx\") pod \"maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:06.529692 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.529629    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:06.560183 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.560150    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:06.802901 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.802784    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:06.804193 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.804171    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"]
Sep 11 04:06:06.806272 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:06:06.806246    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7955d3b7_d5bb_4c4e_8bdf_a4d589499c3f.slice/crio-1c9652ad6d6a07757c27d5831755fdb19a405189433eac7719c5644a7bee0b64 WatchSource:0}: Error finding container 1c9652ad6d6a07757c27d5831755fdb19a405189433eac7719c5644a7bee0b64: Status 404 returned error can't find the container with id 1c9652ad6d6a07757c27d5831755fdb19a405189433eac7719c5644a7bee0b64
Sep 11 04:06:06.808054 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.808012    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 04:06:06.890139 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:06.890104    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:07.593944 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:07.593907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr" event={"ID":"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f","Type":"ContainerStarted","Data":"9615b3687063d49141338a3babdec9f63dd8a7c809c925ec595434ec84762b1e"}
Sep 11 04:06:07.594339 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:07.593973    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:07.594339 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:07.593995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr" event={"ID":"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f","Type":"ContainerStarted","Data":"1c9652ad6d6a07757c27d5831755fdb19a405189433eac7719c5644a7bee0b64"}
Sep 11 04:06:07.615802 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:07.615758    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr" podStartSLOduration=1.1780437 podStartE2EDuration="1.615744083s" totalImagesPullingTime="437.700383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:06:07.613924465 +0000 UTC m=+960.236317520" watchObservedRunningTime="2026-09-11 04:06:07.615744083 +0000 UTC m=+960.238137157"
Sep 11 04:06:09.711635 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:09.711588    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:06:09.712138 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:09.711658    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:06:09.726792 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:09.726763    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:06:10.621437 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:10.621407    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:06:13.605762 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:13.605732    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:14.934270 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:14.934236    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:19.807312 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:19.807271    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"]
Sep 11 04:06:19.825710 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:19.825680    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:20.702647 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:20.702616    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"]
Sep 11 04:06:20.719758 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:20.719725    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:20.838464 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:20.838432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:21.028426 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:21.028339    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:21.137677 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:21.137646    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:22.030289 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:22.030236    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:22.269530 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:22.269492    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:26.160746 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:26.160706    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:27.196064 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:27.196030    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:29.871976 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:29.871937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:30.506062 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:30.506009    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:30.604875 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:30.604843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:30.894303 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:30.894261    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:32.264769 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:32.264728    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:32.949049 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:32.949004    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"]
Sep 11 04:06:32.949263 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:32.949226    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv" podUID="060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a" containerName="maas-api" containerID="cri-o://2edc2314d58f4d226cd013f47ad51496b10a4742ca7d99427f5f6466bc25c5fa" gracePeriod=30
Sep 11 04:06:33.053848 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.053811    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:33.151923 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.151877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:33.200818 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.200765    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:06:33.223775 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.223743    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:33.271475 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.271450    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t6k7\" (UniqueName: \"kubernetes.io/projected/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-kube-api-access-7t6k7\") pod \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") "
Sep 11 04:06:33.271888 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.271533    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-maas-api-tls\") pod \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") "
Sep 11 04:06:33.271888 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.271573    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-openshift-service-ca\") pod \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") "
Sep 11 04:06:33.271888 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.271616    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-metrics-certs\") pod \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\" (UID: \"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a\") "
Sep 11 04:06:33.272099 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.271974    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a" (UID: "060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:06:33.273671 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.273643    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-kube-api-access-7t6k7" (OuterVolumeSpecName: "kube-api-access-7t6k7") pod "060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a" (UID: "060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a"). InnerVolumeSpecName "kube-api-access-7t6k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:06:33.273765 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.273729    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a" (UID: "060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:06:33.273903 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.273889    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a" (UID: "060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:06:33.373045 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.372990    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t6k7\" (UniqueName: \"kubernetes.io/projected/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-kube-api-access-7t6k7\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:33.373045 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.373014    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:33.373045 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.373041    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:33.373045 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.373049    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:33.518247 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.518150    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:33.727374 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.727339    2581 generic.go:356] "Generic (PLEG): container finished" podID="060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a" containerID="2edc2314d58f4d226cd013f47ad51496b10a4742ca7d99427f5f6466bc25c5fa" exitCode=0
Sep 11 04:06:33.727550 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.727386    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"
Sep 11 04:06:33.727550 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.727419    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv" event={"ID":"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a","Type":"ContainerDied","Data":"2edc2314d58f4d226cd013f47ad51496b10a4742ca7d99427f5f6466bc25c5fa"}
Sep 11 04:06:33.727550 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.727481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv" event={"ID":"060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a","Type":"ContainerDied","Data":"c3c54a7e724df85da00668a828d2013eb01d13a458f52447115643b80bcc1af8"}
Sep 11 04:06:33.727550 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.727519    2581 scope.go:117] "RemoveContainer" containerID="2edc2314d58f4d226cd013f47ad51496b10a4742ca7d99427f5f6466bc25c5fa"
Sep 11 04:06:33.737569 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.737554    2581 scope.go:117] "RemoveContainer" containerID="2edc2314d58f4d226cd013f47ad51496b10a4742ca7d99427f5f6466bc25c5fa"
Sep 11 04:06:33.737829 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:06:33.737809    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2edc2314d58f4d226cd013f47ad51496b10a4742ca7d99427f5f6466bc25c5fa\": container with ID starting with 2edc2314d58f4d226cd013f47ad51496b10a4742ca7d99427f5f6466bc25c5fa not found: ID does not exist" containerID="2edc2314d58f4d226cd013f47ad51496b10a4742ca7d99427f5f6466bc25c5fa"
Sep 11 04:06:33.737902 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.737835    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2edc2314d58f4d226cd013f47ad51496b10a4742ca7d99427f5f6466bc25c5fa"} err="failed to get container status \"2edc2314d58f4d226cd013f47ad51496b10a4742ca7d99427f5f6466bc25c5fa\": rpc error: code = NotFound desc = could not find container \"2edc2314d58f4d226cd013f47ad51496b10a4742ca7d99427f5f6466bc25c5fa\": container with ID starting with 2edc2314d58f4d226cd013f47ad51496b10a4742ca7d99427f5f6466bc25c5fa not found: ID does not exist"
Sep 11 04:06:33.748818 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.748796    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"]
Sep 11 04:06:33.750655 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.750635    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-00a938-79d57b7bb4-dnjmv"]
Sep 11 04:06:33.862805 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:33.862776    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a" path="/var/lib/kubelet/pods/060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a/volumes"
Sep 11 04:06:34.827109 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:34.827073    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck" podUID="a11a8c3c-8f83-4283-b8c7-7b6a13991910" containerName="main" containerID="cri-o://9555acce202a157f67888141248c12085d1d8db9d8f51ab231b9084297254d8c" gracePeriod=45
Sep 11 04:06:34.918387 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:34.918348    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck" podUID="a11a8c3c-8f83-4283-b8c7-7b6a13991910" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 11 04:06:35.078592 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.078538    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:06:35.191760 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.191733    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-model-cache\") pod \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") "
Sep 11 04:06:35.191937 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.191792    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj5k9\" (UniqueName: \"kubernetes.io/projected/a11a8c3c-8f83-4283-b8c7-7b6a13991910-kube-api-access-tj5k9\") pod \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") "
Sep 11 04:06:35.191937 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.191815    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-home\") pod \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") "
Sep 11 04:06:35.191937 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.191911    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a11a8c3c-8f83-4283-b8c7-7b6a13991910-tls-certs\") pod \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") "
Sep 11 04:06:35.192119 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.191955    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-dshm\") pod \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") "
Sep 11 04:06:35.192119 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.191985    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-tmp-dir\") pod \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\" (UID: \"a11a8c3c-8f83-4283-b8c7-7b6a13991910\") "
Sep 11 04:06:35.192119 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.191984    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-model-cache" (OuterVolumeSpecName: "model-cache") pod "a11a8c3c-8f83-4283-b8c7-7b6a13991910" (UID: "a11a8c3c-8f83-4283-b8c7-7b6a13991910"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:35.192280 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.192167    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-home" (OuterVolumeSpecName: "home") pod "a11a8c3c-8f83-4283-b8c7-7b6a13991910" (UID: "a11a8c3c-8f83-4283-b8c7-7b6a13991910"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:35.192340 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.192274    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a11a8c3c-8f83-4283-b8c7-7b6a13991910" (UID: "a11a8c3c-8f83-4283-b8c7-7b6a13991910"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:35.192406 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.192390    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-tmp-dir\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:35.192466 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.192407    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-model-cache\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:35.192466 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.192416    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-home\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:35.194103 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.194077    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11a8c3c-8f83-4283-b8c7-7b6a13991910-kube-api-access-tj5k9" (OuterVolumeSpecName: "kube-api-access-tj5k9") pod "a11a8c3c-8f83-4283-b8c7-7b6a13991910" (UID: "a11a8c3c-8f83-4283-b8c7-7b6a13991910"). InnerVolumeSpecName "kube-api-access-tj5k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:06:35.194214 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.194109    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-dshm" (OuterVolumeSpecName: "dshm") pod "a11a8c3c-8f83-4283-b8c7-7b6a13991910" (UID: "a11a8c3c-8f83-4283-b8c7-7b6a13991910"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:35.194214 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.194123    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11a8c3c-8f83-4283-b8c7-7b6a13991910-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a11a8c3c-8f83-4283-b8c7-7b6a13991910" (UID: "a11a8c3c-8f83-4283-b8c7-7b6a13991910"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:06:35.255236 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.255203    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:35.293383 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.293356    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj5k9\" (UniqueName: \"kubernetes.io/projected/a11a8c3c-8f83-4283-b8c7-7b6a13991910-kube-api-access-tj5k9\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:35.293383 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.293381    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a11a8c3c-8f83-4283-b8c7-7b6a13991910-tls-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:35.293513 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.293396    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c3c-8f83-4283-b8c7-7b6a13991910-dshm\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:35.721861 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.721823    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd" podUID="b546c384-3fa8-4b04-9e43-c099f9383930" containerName="main" containerID="cri-o://105d871a562d62907b25148b8886a2a4facbaf0f6b3a94ab88d09b95aae015dc" gracePeriod=45
Sep 11 04:06:35.741218 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.741192    2581 generic.go:356] "Generic (PLEG): container finished" podID="a11a8c3c-8f83-4283-b8c7-7b6a13991910" containerID="9555acce202a157f67888141248c12085d1d8db9d8f51ab231b9084297254d8c" exitCode=0
Sep 11 04:06:35.741339 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.741236    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"
Sep 11 04:06:35.741339 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.741263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck" event={"ID":"a11a8c3c-8f83-4283-b8c7-7b6a13991910","Type":"ContainerDied","Data":"9555acce202a157f67888141248c12085d1d8db9d8f51ab231b9084297254d8c"}
Sep 11 04:06:35.741339 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.741304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck" event={"ID":"a11a8c3c-8f83-4283-b8c7-7b6a13991910","Type":"ContainerDied","Data":"7eadd7786e3406ded52a653d2dc4b3bef920dd6251ce640968dbf1930cef6b79"}
Sep 11 04:06:35.741339 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.741328    2581 scope.go:117] "RemoveContainer" containerID="9555acce202a157f67888141248c12085d1d8db9d8f51ab231b9084297254d8c"
Sep 11 04:06:35.785518 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.785484    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"]
Sep 11 04:06:35.788855 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.788834    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-00a938/test-model-00a938-kserve-5bf7cb7dcf-fvrck"]
Sep 11 04:06:35.833832 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.833732    2581 scope.go:117] "RemoveContainer" containerID="9555acce202a157f67888141248c12085d1d8db9d8f51ab231b9084297254d8c"
Sep 11 04:06:35.834140 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:06:35.834116    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9555acce202a157f67888141248c12085d1d8db9d8f51ab231b9084297254d8c\": container with ID starting with 9555acce202a157f67888141248c12085d1d8db9d8f51ab231b9084297254d8c not found: ID does not exist" containerID="9555acce202a157f67888141248c12085d1d8db9d8f51ab231b9084297254d8c"
Sep 11 04:06:35.834190 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.834150    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9555acce202a157f67888141248c12085d1d8db9d8f51ab231b9084297254d8c"} err="failed to get container status \"9555acce202a157f67888141248c12085d1d8db9d8f51ab231b9084297254d8c\": rpc error: code = NotFound desc = could not find container \"9555acce202a157f67888141248c12085d1d8db9d8f51ab231b9084297254d8c\": container with ID starting with 9555acce202a157f67888141248c12085d1d8db9d8f51ab231b9084297254d8c not found: ID does not exist"
Sep 11 04:06:35.863669 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.863639    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a11a8c3c-8f83-4283-b8c7-7b6a13991910" path="/var/lib/kubelet/pods/a11a8c3c-8f83-4283-b8c7-7b6a13991910/volumes"
Sep 11 04:06:35.978711 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:35.978653    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:06:36.100226 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.100196    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fb8x\" (UniqueName: \"kubernetes.io/projected/b546c384-3fa8-4b04-9e43-c099f9383930-kube-api-access-6fb8x\") pod \"b546c384-3fa8-4b04-9e43-c099f9383930\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") "
Sep 11 04:06:36.100356 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.100258    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-dshm\") pod \"b546c384-3fa8-4b04-9e43-c099f9383930\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") "
Sep 11 04:06:36.100356 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.100288    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-tmp-dir\") pod \"b546c384-3fa8-4b04-9e43-c099f9383930\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") "
Sep 11 04:06:36.100356 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.100323    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b546c384-3fa8-4b04-9e43-c099f9383930-tls-certs\") pod \"b546c384-3fa8-4b04-9e43-c099f9383930\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") "
Sep 11 04:06:36.100356 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.100344    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-home\") pod \"b546c384-3fa8-4b04-9e43-c099f9383930\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") "
Sep 11 04:06:36.100577 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.100382    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-model-cache\") pod \"b546c384-3fa8-4b04-9e43-c099f9383930\" (UID: \"b546c384-3fa8-4b04-9e43-c099f9383930\") "
Sep 11 04:06:36.100577 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.100511    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b546c384-3fa8-4b04-9e43-c099f9383930" (UID: "b546c384-3fa8-4b04-9e43-c099f9383930"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:36.100686 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.100623    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-home" (OuterVolumeSpecName: "home") pod "b546c384-3fa8-4b04-9e43-c099f9383930" (UID: "b546c384-3fa8-4b04-9e43-c099f9383930"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:36.100686 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.100659    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-model-cache" (OuterVolumeSpecName: "model-cache") pod "b546c384-3fa8-4b04-9e43-c099f9383930" (UID: "b546c384-3fa8-4b04-9e43-c099f9383930"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:36.102556 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.102534    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-dshm" (OuterVolumeSpecName: "dshm") pod "b546c384-3fa8-4b04-9e43-c099f9383930" (UID: "b546c384-3fa8-4b04-9e43-c099f9383930"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:36.102635 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.102548    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b546c384-3fa8-4b04-9e43-c099f9383930-kube-api-access-6fb8x" (OuterVolumeSpecName: "kube-api-access-6fb8x") pod "b546c384-3fa8-4b04-9e43-c099f9383930" (UID: "b546c384-3fa8-4b04-9e43-c099f9383930"). InnerVolumeSpecName "kube-api-access-6fb8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:06:36.102635 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.102562    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b546c384-3fa8-4b04-9e43-c099f9383930-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b546c384-3fa8-4b04-9e43-c099f9383930" (UID: "b546c384-3fa8-4b04-9e43-c099f9383930"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:06:36.201583 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.201547    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-tmp-dir\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:36.201583 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.201577    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b546c384-3fa8-4b04-9e43-c099f9383930-tls-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:36.201583 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.201585    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-home\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:36.201583 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.201593    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-model-cache\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:36.201827 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.201603    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fb8x\" (UniqueName: \"kubernetes.io/projected/b546c384-3fa8-4b04-9e43-c099f9383930-kube-api-access-6fb8x\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:36.201827 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.201610    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b546c384-3fa8-4b04-9e43-c099f9383930-dshm\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:36.609575 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.609535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:36.710417 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.710385    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:36.748569 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.748537    2581 generic.go:356] "Generic (PLEG): container finished" podID="b546c384-3fa8-4b04-9e43-c099f9383930" containerID="105d871a562d62907b25148b8886a2a4facbaf0f6b3a94ab88d09b95aae015dc" exitCode=0
Sep 11 04:06:36.748719 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.748584    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"
Sep 11 04:06:36.748719 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.748627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd" event={"ID":"b546c384-3fa8-4b04-9e43-c099f9383930","Type":"ContainerDied","Data":"105d871a562d62907b25148b8886a2a4facbaf0f6b3a94ab88d09b95aae015dc"}
Sep 11 04:06:36.748719 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.748668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd" event={"ID":"b546c384-3fa8-4b04-9e43-c099f9383930","Type":"ContainerDied","Data":"e6e3fd1ef97ead9b25608dbbd9487726b9d5911e2f3adccf80598722db7e2b81"}
Sep 11 04:06:36.748719 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.748691    2581 scope.go:117] "RemoveContainer" containerID="105d871a562d62907b25148b8886a2a4facbaf0f6b3a94ab88d09b95aae015dc"
Sep 11 04:06:36.757718 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.757691    2581 scope.go:117] "RemoveContainer" containerID="105d871a562d62907b25148b8886a2a4facbaf0f6b3a94ab88d09b95aae015dc"
Sep 11 04:06:36.757972 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:06:36.757953    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105d871a562d62907b25148b8886a2a4facbaf0f6b3a94ab88d09b95aae015dc\": container with ID starting with 105d871a562d62907b25148b8886a2a4facbaf0f6b3a94ab88d09b95aae015dc not found: ID does not exist" containerID="105d871a562d62907b25148b8886a2a4facbaf0f6b3a94ab88d09b95aae015dc"
Sep 11 04:06:36.758086 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.757979    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105d871a562d62907b25148b8886a2a4facbaf0f6b3a94ab88d09b95aae015dc"} err="failed to get container status \"105d871a562d62907b25148b8886a2a4facbaf0f6b3a94ab88d09b95aae015dc\": rpc error: code = NotFound desc = could not find container \"105d871a562d62907b25148b8886a2a4facbaf0f6b3a94ab88d09b95aae015dc\": container with ID starting with 105d871a562d62907b25148b8886a2a4facbaf0f6b3a94ab88d09b95aae015dc not found: ID does not exist"
Sep 11 04:06:36.769606 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.769585    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"]
Sep 11 04:06:36.773313 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:36.773292    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ce78b8/noten-model-ce78b8-kserve-769986bcdc-g2gbd"]
Sep 11 04:06:37.043824 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:37.043738    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:37.862248 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:37.862217    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b546c384-3fa8-4b04-9e43-c099f9383930" path="/var/lib/kubelet/pods/b546c384-3fa8-4b04-9e43-c099f9383930/volumes"
Sep 11 04:06:39.498291 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:39.498253    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:39.556829 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:39.556798    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:39.740384 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:39.740356    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:40.241309 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:40.241275    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:42.843526 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:42.843485    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:47.069920 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:47.069884    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:47.612202 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:47.612171    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"]
Sep 11 04:06:47.612402 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:47.612379    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr" podUID="7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f" containerName="maas-api" containerID="cri-o://9615b3687063d49141338a3babdec9f63dd8a7c809c925ec595434ec84762b1e" gracePeriod=30
Sep 11 04:06:47.639532 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:47.639341    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:47.738232 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:47.738204    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:47.810427 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:47.810398    2581 generic.go:356] "Generic (PLEG): container finished" podID="7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f" containerID="9615b3687063d49141338a3babdec9f63dd8a7c809c925ec595434ec84762b1e" exitCode=0
Sep 11 04:06:47.810566 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:47.810455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr" event={"ID":"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f","Type":"ContainerDied","Data":"9615b3687063d49141338a3babdec9f63dd8a7c809c925ec595434ec84762b1e"}
Sep 11 04:06:47.829158 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:47.829127    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:47.864284 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:47.864225    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:47.885329 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:47.885287    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:48.013354 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.013325    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-maas-api-tls\") pod \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") "
Sep 11 04:06:48.013509 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.013361    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-openshift-service-ca\") pod \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") "
Sep 11 04:06:48.013509 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.013385    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khfwx\" (UniqueName: \"kubernetes.io/projected/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-kube-api-access-khfwx\") pod \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") "
Sep 11 04:06:48.013509 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.013412    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-metrics-certs\") pod \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\" (UID: \"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f\") "
Sep 11 04:06:48.014007 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.013780    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f" (UID: "7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:06:48.014007 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.013877    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:48.015503 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.015474    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f" (UID: "7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:06:48.015608 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.015565    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-kube-api-access-khfwx" (OuterVolumeSpecName: "kube-api-access-khfwx") pod "7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f" (UID: "7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f"). InnerVolumeSpecName "kube-api-access-khfwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:06:48.015608 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.015567    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f" (UID: "7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:06:48.114741 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.114669    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:48.114741 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.114692    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khfwx\" (UniqueName: \"kubernetes.io/projected/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-kube-api-access-khfwx\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:48.114741 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.114702    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:48.179752 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.179719    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:48.825955 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.825927    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"
Sep 11 04:06:48.826130 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.825954    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr" event={"ID":"7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f","Type":"ContainerDied","Data":"1c9652ad6d6a07757c27d5831755fdb19a405189433eac7719c5644a7bee0b64"}
Sep 11 04:06:48.826130 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.826065    2581 scope.go:117] "RemoveContainer" containerID="9615b3687063d49141338a3babdec9f63dd8a7c809c925ec595434ec84762b1e"
Sep 11 04:06:48.847825 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.847789    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"]
Sep 11 04:06:48.851561 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:48.851540    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d69ee02-6f754d7cdd-n2jgr"]
Sep 11 04:06:49.861045 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:49.860983    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f" path="/var/lib/kubelet/pods/7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f/volumes"
Sep 11 04:06:50.206601 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:50.206507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:50.762009 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:50.761978    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"]
Sep 11 04:06:50.762303 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:50.762275    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz" podUID="2bdd0cfa-7899-4c06-a7c8-260406688630" containerName="maas-api" containerID="cri-o://7873b7f7e68289eb5156d7f31254a72bce19f3d16cac87d56de18aaf1d963140" gracePeriod=30
Sep 11 04:06:50.785348 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:50.785317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:50.850453 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:50.850422    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:50.954402 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:50.954366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:51.052162 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.052099    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:51.216002 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.215975    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:06:51.292487 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.292402    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:51.345394 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.345362    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-maas-api-tls\") pod \"2bdd0cfa-7899-4c06-a7c8-260406688630\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") "
Sep 11 04:06:51.345394 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.345397    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdd0cfa-7899-4c06-a7c8-260406688630-openshift-service-ca\") pod \"2bdd0cfa-7899-4c06-a7c8-260406688630\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") "
Sep 11 04:06:51.345613 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.345467    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9lsj\" (UniqueName: \"kubernetes.io/projected/2bdd0cfa-7899-4c06-a7c8-260406688630-kube-api-access-v9lsj\") pod \"2bdd0cfa-7899-4c06-a7c8-260406688630\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") "
Sep 11 04:06:51.345613 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.345527    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-metrics-certs\") pod \"2bdd0cfa-7899-4c06-a7c8-260406688630\" (UID: \"2bdd0cfa-7899-4c06-a7c8-260406688630\") "
Sep 11 04:06:51.345792 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.345763    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdd0cfa-7899-4c06-a7c8-260406688630-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2bdd0cfa-7899-4c06-a7c8-260406688630" (UID: "2bdd0cfa-7899-4c06-a7c8-260406688630"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:06:51.347722 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.347686    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdd0cfa-7899-4c06-a7c8-260406688630-kube-api-access-v9lsj" (OuterVolumeSpecName: "kube-api-access-v9lsj") pod "2bdd0cfa-7899-4c06-a7c8-260406688630" (UID: "2bdd0cfa-7899-4c06-a7c8-260406688630"). InnerVolumeSpecName "kube-api-access-v9lsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:06:51.347722 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.347709    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2bdd0cfa-7899-4c06-a7c8-260406688630" (UID: "2bdd0cfa-7899-4c06-a7c8-260406688630"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:06:51.347872 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.347729    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2bdd0cfa-7899-4c06-a7c8-260406688630" (UID: "2bdd0cfa-7899-4c06-a7c8-260406688630"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:06:51.446899 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.446865    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9lsj\" (UniqueName: \"kubernetes.io/projected/2bdd0cfa-7899-4c06-a7c8-260406688630-kube-api-access-v9lsj\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:51.446899 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.446893    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:51.446899 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.446903    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdd0cfa-7899-4c06-a7c8-260406688630-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:51.447174 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.446911    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdd0cfa-7899-4c06-a7c8-260406688630-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:06:51.843310 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.843279    2581 generic.go:356] "Generic (PLEG): container finished" podID="2bdd0cfa-7899-4c06-a7c8-260406688630" containerID="7873b7f7e68289eb5156d7f31254a72bce19f3d16cac87d56de18aaf1d963140" exitCode=0
Sep 11 04:06:51.843489 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.843323    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"
Sep 11 04:06:51.843489 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.843372    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz" event={"ID":"2bdd0cfa-7899-4c06-a7c8-260406688630","Type":"ContainerDied","Data":"7873b7f7e68289eb5156d7f31254a72bce19f3d16cac87d56de18aaf1d963140"}
Sep 11 04:06:51.843489 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.843421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz" event={"ID":"2bdd0cfa-7899-4c06-a7c8-260406688630","Type":"ContainerDied","Data":"8d441936a0cd03a0c913418b0f62ab4e07df7ef5f008dc5ea5f6abd76a548b44"}
Sep 11 04:06:51.843489 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.843443    2581 scope.go:117] "RemoveContainer" containerID="7873b7f7e68289eb5156d7f31254a72bce19f3d16cac87d56de18aaf1d963140"
Sep 11 04:06:51.852781 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.852766    2581 scope.go:117] "RemoveContainer" containerID="7873b7f7e68289eb5156d7f31254a72bce19f3d16cac87d56de18aaf1d963140"
Sep 11 04:06:51.853054 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:06:51.853031    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7873b7f7e68289eb5156d7f31254a72bce19f3d16cac87d56de18aaf1d963140\": container with ID starting with 7873b7f7e68289eb5156d7f31254a72bce19f3d16cac87d56de18aaf1d963140 not found: ID does not exist" containerID="7873b7f7e68289eb5156d7f31254a72bce19f3d16cac87d56de18aaf1d963140"
Sep 11 04:06:51.853108 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.853065    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7873b7f7e68289eb5156d7f31254a72bce19f3d16cac87d56de18aaf1d963140"} err="failed to get container status \"7873b7f7e68289eb5156d7f31254a72bce19f3d16cac87d56de18aaf1d963140\": rpc error: code = NotFound desc = could not find container \"7873b7f7e68289eb5156d7f31254a72bce19f3d16cac87d56de18aaf1d963140\": container with ID starting with 7873b7f7e68289eb5156d7f31254a72bce19f3d16cac87d56de18aaf1d963140 not found: ID does not exist"
Sep 11 04:06:51.863862 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.863829    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"]
Sep 11 04:06:51.873232 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:51.873208    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d685d-84bbcdb8bd-w7khz"]
Sep 11 04:06:53.860716 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:53.860684    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bdd0cfa-7899-4c06-a7c8-260406688630" path="/var/lib/kubelet/pods/2bdd0cfa-7899-4c06-a7c8-260406688630/volumes"
Sep 11 04:06:54.068980 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:54.068947    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:54.140317 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:54.140228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:54.308342 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:54.308306    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:54.649880 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:54.649842    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:54.755310 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:54.755281    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:54.857576 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:54.857541    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:57.098845 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:57.098802    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:57.199226 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:57.199193    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:57.320167 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:57.320121    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:57.827745 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:57.827705    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:58.144543 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:58.144454    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:06:58.328120 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:06:58.328083    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:00.595565 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:00.595529    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:00.696126 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:00.696092    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:00.797667 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:00.797635    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:00.924722 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:00.924642    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:01.176535 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:01.176452    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:01.274205 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:01.274173    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:02.505686 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.505652    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"]
Sep 11 04:07:02.506132 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506113    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a" containerName="maas-api"
Sep 11 04:07:02.506132 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506134    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a" containerName="maas-api"
Sep 11 04:07:02.506217 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506146    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f" containerName="maas-api"
Sep 11 04:07:02.506217 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506152    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f" containerName="maas-api"
Sep 11 04:07:02.506217 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506160    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b546c384-3fa8-4b04-9e43-c099f9383930" containerName="main"
Sep 11 04:07:02.506217 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506166    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b546c384-3fa8-4b04-9e43-c099f9383930" containerName="main"
Sep 11 04:07:02.506217 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506181    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bdd0cfa-7899-4c06-a7c8-260406688630" containerName="maas-api"
Sep 11 04:07:02.506217 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506186    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bdd0cfa-7899-4c06-a7c8-260406688630" containerName="maas-api"
Sep 11 04:07:02.506217 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506195    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a11a8c3c-8f83-4283-b8c7-7b6a13991910" containerName="main"
Sep 11 04:07:02.506217 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506200    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a11a8c3c-8f83-4283-b8c7-7b6a13991910" containerName="main"
Sep 11 04:07:02.506461 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506264    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bdd0cfa-7899-4c06-a7c8-260406688630" containerName="maas-api"
Sep 11 04:07:02.506461 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506272    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="060eb9dd-78ae-4b3e-be8e-1d83c2b48b8a" containerName="maas-api"
Sep 11 04:07:02.506461 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506277    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a11a8c3c-8f83-4283-b8c7-7b6a13991910" containerName="main"
Sep 11 04:07:02.506461 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506286    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b546c384-3fa8-4b04-9e43-c099f9383930" containerName="main"
Sep 11 04:07:02.506461 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.506291    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7955d3b7-d5bb-4c4e-8bdf-a4d589499c3f" containerName="maas-api"
Sep 11 04:07:02.511223 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.511202    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:02.514661 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.514638    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 04:07:02.514836 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.514682    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wb8bv\""
Sep 11 04:07:02.515007 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.514990    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 04:07:02.515135 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.515038    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-9d031d-serving-cert\""
Sep 11 04:07:02.515135 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.515095    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:07:02.520705 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.520683    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"]
Sep 11 04:07:02.550341 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.550298    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:02.553494 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.553469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf4pb\" (UniqueName: \"kubernetes.io/projected/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-kube-api-access-vf4pb\") pod \"maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:02.553617 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.553512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:02.553617 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.553548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:02.553617 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.553597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:02.648758 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.648720    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:02.654974 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.654954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf4pb\" (UniqueName: \"kubernetes.io/projected/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-kube-api-access-vf4pb\") pod \"maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:02.655061 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.654986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:02.655061 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.655008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:02.655161 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.655066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:02.655205 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:07:02.655184    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-serving-cert: secret "maas-api-e2e-shared-a-w3-9d031d-serving-cert" not found
Sep 11 04:07:02.655266 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:07:02.655256    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-maas-api-tls podName:d1364e0a-a2f6-4c5a-8b2d-65d63b3db641 nodeName:}" failed. No retries permitted until 2026-09-11 04:07:03.155241932 +0000 UTC m=+1015.777634942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-maas-api-tls") pod "maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l" (UID: "d1364e0a-a2f6-4c5a-8b2d-65d63b3db641") : secret "maas-api-e2e-shared-a-w3-9d031d-serving-cert" not found
Sep 11 04:07:02.655651 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.655629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:02.657488 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.657466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:02.668435 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.668415    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf4pb\" (UniqueName: \"kubernetes.io/projected/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-kube-api-access-vf4pb\") pod \"maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:02.833165 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:02.833126    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:03.159744 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:03.159657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:03.162010 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:03.161984    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:03.195701 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:03.195667    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:03.303447 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:03.303389    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:03.423972 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:03.423892    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:03.701303 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:03.701276    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"]
Sep 11 04:07:03.702991 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:07:03.702958    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1364e0a_a2f6_4c5a_8b2d_65d63b3db641.slice/crio-7d1834434dc81a7a8e7ab87a72cc10f1f2b3f53acd4261c761916950a88c147a WatchSource:0}: Error finding container 7d1834434dc81a7a8e7ab87a72cc10f1f2b3f53acd4261c761916950a88c147a: Status 404 returned error can't find the container with id 7d1834434dc81a7a8e7ab87a72cc10f1f2b3f53acd4261c761916950a88c147a
Sep 11 04:07:03.903878 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:03.903844    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l" event={"ID":"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641","Type":"ContainerStarted","Data":"7d1834434dc81a7a8e7ab87a72cc10f1f2b3f53acd4261c761916950a88c147a"}
Sep 11 04:07:04.245449 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:04.245417    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:04.910611 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:04.910575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l" event={"ID":"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641","Type":"ContainerStarted","Data":"514cafea76a8557b153316a5c39958b03bdf9a7a0bb28b82ed1f76fee9c2c244"}
Sep 11 04:07:04.910996 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:04.910662    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:04.930674 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:04.930625    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l" podStartSLOduration=2.472962225 podStartE2EDuration="2.930610389s" totalImagesPullingTime="457.648164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:04.928563694 +0000 UTC m=+1017.550956747" watchObservedRunningTime="2026-09-11 04:07:04.930610389 +0000 UTC m=+1017.553003416"
Sep 11 04:07:06.681131 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:06.681086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:06.781977 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:06.781935    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:07.561844 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:07.561806    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:07.832920 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:07.832832    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:08.115197 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:08.115105    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:10.924148 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:10.924114    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:07:11.380531 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:11.380470    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:11.478004 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:11.477965    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:12.509505 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.509464    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"]
Sep 11 04:07:12.514773 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.514749    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:12.517373 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.517349    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-68ec26-serving-cert\""
Sep 11 04:07:12.525400 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.525371    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"]
Sep 11 04:07:12.545889 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.545861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbbpl\" (UniqueName: \"kubernetes.io/projected/9ebe43be-27d5-45d6-82ff-192bf0879aeb-kube-api-access-cbbpl\") pod \"maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:12.546067 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.545893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebe43be-27d5-45d6-82ff-192bf0879aeb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:12.546067 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.545934    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:12.546067 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.546006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:12.596619 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.596581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:12.647311 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.647284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:12.647462 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.647321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:12.647462 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.647386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbbpl\" (UniqueName: \"kubernetes.io/projected/9ebe43be-27d5-45d6-82ff-192bf0879aeb-kube-api-access-cbbpl\") pod \"maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:12.647462 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.647407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebe43be-27d5-45d6-82ff-192bf0879aeb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:12.647655 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:07:12.647489    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-serving-cert: secret "maas-api-e2e-shared-a-w2-68ec26-serving-cert" not found
Sep 11 04:07:12.647655 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:07:12.647561    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-maas-api-tls podName:9ebe43be-27d5-45d6-82ff-192bf0879aeb nodeName:}" failed. No retries permitted until 2026-09-11 04:07:13.147547381 +0000 UTC m=+1025.769940385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-maas-api-tls") pod "maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd" (UID: "9ebe43be-27d5-45d6-82ff-192bf0879aeb") : secret "maas-api-e2e-shared-a-w2-68ec26-serving-cert" not found
Sep 11 04:07:12.648047 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.648014    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebe43be-27d5-45d6-82ff-192bf0879aeb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:12.649640 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.649625    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:12.655837 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.655820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbbpl\" (UniqueName: \"kubernetes.io/projected/9ebe43be-27d5-45d6-82ff-192bf0879aeb-kube-api-access-cbbpl\") pod \"maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:12.699361 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.699338    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:12.869164 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:12.869133    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:13.058257 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:13.058227    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:13.152429 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:13.152351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:13.154626 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:13.154606    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:13.163268 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:13.163235    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:13.426898 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:13.426809    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:13.709506 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:13.709480    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"]
Sep 11 04:07:13.711399 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:07:13.711372    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebe43be_27d5_45d6_82ff_192bf0879aeb.slice/crio-48b21957f556faa1d2260a7c17b4cc0b7a7e6257f264f3044e0c914fb5de4d36 WatchSource:0}: Error finding container 48b21957f556faa1d2260a7c17b4cc0b7a7e6257f264f3044e0c914fb5de4d36: Status 404 returned error can't find the container with id 48b21957f556faa1d2260a7c17b4cc0b7a7e6257f264f3044e0c914fb5de4d36
Sep 11 04:07:13.959817 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:13.959739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd" event={"ID":"9ebe43be-27d5-45d6-82ff-192bf0879aeb","Type":"ContainerStarted","Data":"48b21957f556faa1d2260a7c17b4cc0b7a7e6257f264f3044e0c914fb5de4d36"}
Sep 11 04:07:14.967301 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:14.967265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd" event={"ID":"9ebe43be-27d5-45d6-82ff-192bf0879aeb","Type":"ContainerStarted","Data":"9e185d1632a888d3a5f2531b3ca38a6490814627931e4c8e10f364274fbbf2ca"}
Sep 11 04:07:14.967705 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:14.967347    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:14.983652 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:14.983604    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd" podStartSLOduration=2.552491677 podStartE2EDuration="2.983588005s" totalImagesPullingTime="431.096328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:14.982842415 +0000 UTC m=+1027.605235459" watchObservedRunningTime="2026-09-11 04:07:14.983588005 +0000 UTC m=+1027.605981032"
Sep 11 04:07:20.977758 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:20.977729    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:07:23.639286 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:23.639251    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:26.019257 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:26.019213    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:26.330065 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:26.330029    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:26.660841 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:26.660751    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:26.895263 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:26.895221    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:29.658593 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:29.658554    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:29.754550 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:29.754512    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:30.896780 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:30.896747    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"]
Sep 11 04:07:30.900347 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:30.900322    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:30.903342 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:30.903318    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-c158b2-serving-cert\""
Sep 11 04:07:30.914670 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:30.914647    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"]
Sep 11 04:07:30.957477 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:30.957423    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:31.022520 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.022475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec77937-b130-40f0-865d-aa13509e09e2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:31.022709 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.022564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhf2x\" (UniqueName: \"kubernetes.io/projected/5ec77937-b130-40f0-865d-aa13509e09e2-kube-api-access-mhf2x\") pod \"maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:31.022709 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.022672    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ec77937-b130-40f0-865d-aa13509e09e2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:31.022800 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.022725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec77937-b130-40f0-865d-aa13509e09e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:31.057731 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.057700    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:31.123964 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.123934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ec77937-b130-40f0-865d-aa13509e09e2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:31.124143 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.123971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec77937-b130-40f0-865d-aa13509e09e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:31.124143 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.124000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec77937-b130-40f0-865d-aa13509e09e2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:31.124143 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.124063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhf2x\" (UniqueName: \"kubernetes.io/projected/5ec77937-b130-40f0-865d-aa13509e09e2-kube-api-access-mhf2x\") pod \"maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:31.124747 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.124727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec77937-b130-40f0-865d-aa13509e09e2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:31.126377 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.126350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec77937-b130-40f0-865d-aa13509e09e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:31.126492 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.126439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ec77937-b130-40f0-865d-aa13509e09e2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:31.132119 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.132093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhf2x\" (UniqueName: \"kubernetes.io/projected/5ec77937-b130-40f0-865d-aa13509e09e2-kube-api-access-mhf2x\") pod \"maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:31.157169 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.157098    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:31.211430 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.211401    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:31.498454 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.498427    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"]
Sep 11 04:07:31.500525 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:07:31.500497    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec77937_b130_40f0_865d_aa13509e09e2.slice/crio-46c268b0107acb4176ed8ffb213380cea1f84fa3738f26f10435602cf872cfaf WatchSource:0}: Error finding container 46c268b0107acb4176ed8ffb213380cea1f84fa3738f26f10435602cf872cfaf: Status 404 returned error can't find the container with id 46c268b0107acb4176ed8ffb213380cea1f84fa3738f26f10435602cf872cfaf
Sep 11 04:07:31.599647 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.599590    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:31.716386 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:31.716354    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:32.063612 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:32.063574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh" event={"ID":"5ec77937-b130-40f0-865d-aa13509e09e2","Type":"ContainerStarted","Data":"88f3ec39e789c8233dc12c8eda4c0817a5a17a0525d9d8c10aac116192bfcf9e"}
Sep 11 04:07:32.064001 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:32.063635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh" event={"ID":"5ec77937-b130-40f0-865d-aa13509e09e2","Type":"ContainerStarted","Data":"46c268b0107acb4176ed8ffb213380cea1f84fa3738f26f10435602cf872cfaf"}
Sep 11 04:07:32.064001 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:32.063774    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:32.085701 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:32.085656    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh" podStartSLOduration=1.692726533 podStartE2EDuration="2.085643826s" totalImagesPullingTime="392.917293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:32.084172332 +0000 UTC m=+1044.706565379" watchObservedRunningTime="2026-09-11 04:07:32.085643826 +0000 UTC m=+1044.708036853"
Sep 11 04:07:34.413196 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:34.413154    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:36.290626 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:36.290582    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:36.611739 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:36.611700    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:36.906155 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:36.906046    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:37.471926 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:37.471889    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:38.073524 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:38.073495    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:07:39.519387 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:39.519347    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:39.610149 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:39.610114    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:40.756200 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.756169    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"]
Sep 11 04:07:40.760517 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.760492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:40.769533 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.763703    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-604323-serving-cert\""
Sep 11 04:07:40.776724 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.776698    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"]
Sep 11 04:07:40.800565 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.800527    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:40.818999 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.818968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:40.819140 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.819079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:40.819140 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.819103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:40.819229 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.819157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f595j\" (UniqueName: \"kubernetes.io/projected/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-kube-api-access-f595j\") pod \"maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:40.900686 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.900654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:40.919834 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.919802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:40.919974 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:07:40.919951    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-serving-cert: secret "maas-api-e2e-shared-b-w2-604323-serving-cert" not found
Sep 11 04:07:40.920054 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:07:40.920046    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-maas-api-tls podName:ba6220da-66a5-4eb4-9654-9fba8a4c84e3 nodeName:}" failed. No retries permitted until 2026-09-11 04:07:41.419999018 +0000 UTC m=+1054.042392043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-maas-api-tls") pod "maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g" (UID: "ba6220da-66a5-4eb4-9654-9fba8a4c84e3") : secret "maas-api-e2e-shared-b-w2-604323-serving-cert" not found
Sep 11 04:07:40.920181 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.920161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:40.920243 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.920214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:40.920296 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.920247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f595j\" (UniqueName: \"kubernetes.io/projected/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-kube-api-access-f595j\") pod \"maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:40.921163 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.921141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:40.923260 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.923237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:40.932620 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:40.932600    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f595j\" (UniqueName: \"kubernetes.io/projected/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-kube-api-access-f595j\") pod \"maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:41.222288 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:41.222244    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:41.331637 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:41.331599    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:41.422506 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:41.422471    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:41.424462 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:41.424434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:41.426820 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:41.426794    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:41.675964 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:41.675928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:41.946429 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:41.946400    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"]
Sep 11 04:07:41.948430 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:07:41.948408    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6220da_66a5_4eb4_9654_9fba8a4c84e3.slice/crio-a62fad04bf6aeb0339013273d8823fe0eb127b97f7ab23b369c84354001712a4 WatchSource:0}: Error finding container a62fad04bf6aeb0339013273d8823fe0eb127b97f7ab23b369c84354001712a4: Status 404 returned error can't find the container with id a62fad04bf6aeb0339013273d8823fe0eb127b97f7ab23b369c84354001712a4
Sep 11 04:07:42.116363 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:42.116328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g" event={"ID":"ba6220da-66a5-4eb4-9654-9fba8a4c84e3","Type":"ContainerStarted","Data":"a62fad04bf6aeb0339013273d8823fe0eb127b97f7ab23b369c84354001712a4"}
Sep 11 04:07:43.122584 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:43.122541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g" event={"ID":"ba6220da-66a5-4eb4-9654-9fba8a4c84e3","Type":"ContainerStarted","Data":"5dc8e5217e1c03643703828897dbaa50a13a45f7c48055588fef148fed669800"}
Sep 11 04:07:43.122969 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:43.122609    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:43.140657 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:43.140606    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g" podStartSLOduration=2.781037602 podStartE2EDuration="3.14059151s" totalImagesPullingTime="359.553908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:43.138526157 +0000 UTC m=+1055.760919186" watchObservedRunningTime="2026-09-11 04:07:43.14059151 +0000 UTC m=+1055.762984537"
Sep 11 04:07:49.133627 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:49.133596    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:07:52.372653 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:52.372611    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:54.493724 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:54.493683    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:07:59.097487 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:07:59.097453    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:02.864116 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:02.864036    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:04.474841 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:04.474805    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:08.295214 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.295174    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"]
Sep 11 04:08:08.299602 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.299568    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.302472 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.302447    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9d031d\"/\"kube-root-ca.crt\""
Sep 11 04:08:08.303747 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.303721    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9d031d\"/\"openshift-service-ca.crt\""
Sep 11 04:08:08.303888 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.303782    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9d031d\"/\"default-dockercfg-x45hg\""
Sep 11 04:08:08.304039 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.303994    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9d031d\"/\"auth-test-model-9d031d-kserve-self-signed-certs\""
Sep 11 04:08:08.309124 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.309099    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"]
Sep 11 04:08:08.331858 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.331824    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:08.381184 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.381147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ldc7\" (UniqueName: \"kubernetes.io/projected/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-kube-api-access-6ldc7\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.381184 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.381185    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-dshm\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.381372 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.381201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-model-cache\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.381372 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.381318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-tls-certs\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.381445 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.381374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-home\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.381445 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.381417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-tmp-dir\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.481942 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.481905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-tls-certs\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.482124 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.481981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-home\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.482124 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.482003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-tmp-dir\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.482124 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.482089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ldc7\" (UniqueName: \"kubernetes.io/projected/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-kube-api-access-6ldc7\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.482124 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.482106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-dshm\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.482124 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.482121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-model-cache\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.482500 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.482479    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-home\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.482594 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.482518    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-model-cache\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.482594 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.482536    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-tmp-dir\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.484358 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.484335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-dshm\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.484439 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.484405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-tls-certs\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.491255 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.491235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ldc7\" (UniqueName: \"kubernetes.io/projected/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-kube-api-access-6ldc7\") pod \"auth-test-model-9d031d-kserve-5b79d66d67-c64fn\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.610715 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.610676    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:08.897509 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:08.897484    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"]
Sep 11 04:08:08.898601 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:08:08.898575    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0e28c26_3f03_4c29_a0f1_c752cf0a6d3d.slice/crio-56e74b33d1338dd72ab07e8af451e63039986673dea5331db5ef5bad8382f249 WatchSource:0}: Error finding container 56e74b33d1338dd72ab07e8af451e63039986673dea5331db5ef5bad8382f249: Status 404 returned error can't find the container with id 56e74b33d1338dd72ab07e8af451e63039986673dea5331db5ef5bad8382f249
Sep 11 04:08:09.260277 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:09.260194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn" event={"ID":"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d","Type":"ContainerStarted","Data":"6b93e421a9a7f355323d68f76fcd75c96caa68cc22d4d1140a0f7d45fee2830f"}
Sep 11 04:08:09.260277 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:09.260233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn" event={"ID":"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d","Type":"ContainerStarted","Data":"56e74b33d1338dd72ab07e8af451e63039986673dea5331db5ef5bad8382f249"}
Sep 11 04:08:09.277032 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:09.276975    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn" podStartSLOduration=1.276958988 podStartE2EDuration="1.276958988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:08:09.275656962 +0000 UTC m=+1081.898050021" watchObservedRunningTime="2026-09-11 04:08:09.276958988 +0000 UTC m=+1081.899352016"
Sep 11 04:08:14.061804 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:14.061769    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:14.371664 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:14.371573    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:15.255139 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:15.255108    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:17.625361 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:17.625319    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:17.726416 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:17.726368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:18.611554 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.611513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:18.611769 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.611582    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:18.624751 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.624718    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:18.757956 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.757926    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"]
Sep 11 04:08:18.762081 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.762058    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:18.764784 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.764766    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0347a9fa-serving-cert\""
Sep 11 04:08:18.773398 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.773374    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"]
Sep 11 04:08:18.796448 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.796409    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:18.886776 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.886681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-openshift-service-ca\") pod \"maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:18.886776 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.886748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-metrics-certs\") pod \"maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:18.886970 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.886785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2sg\" (UniqueName: \"kubernetes.io/projected/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-kube-api-access-8d2sg\") pod \"maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:18.886970 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.886842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-maas-api-tls\") pod \"maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:18.896166 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.896137    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:18.988421 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.988391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-metrics-certs\") pod \"maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:18.988585 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.988433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2sg\" (UniqueName: \"kubernetes.io/projected/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-kube-api-access-8d2sg\") pod \"maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:18.988585 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.988468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-maas-api-tls\") pod \"maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:18.988585 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.988536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-openshift-service-ca\") pod \"maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:18.989224 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.989197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-openshift-service-ca\") pod \"maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:18.991053 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.991001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-metrics-certs\") pod \"maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:18.991053 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.991004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-maas-api-tls\") pod \"maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:18.996835 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.996799    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:18.997765 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:18.997747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2sg\" (UniqueName: \"kubernetes.io/projected/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-kube-api-access-8d2sg\") pod \"maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:19.075033 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:19.075001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:19.331541 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:19.331513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:08:19.359668 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:19.359640    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"]
Sep 11 04:08:19.360435 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:08:19.360403    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1362c9_f4d8_4a41_99eb_b6cec8bb8f94.slice/crio-928462429eb3d186aac0ab7c7e520e06e91f8b4fd7bb8faf1f5dbc669a741252 WatchSource:0}: Error finding container 928462429eb3d186aac0ab7c7e520e06e91f8b4fd7bb8faf1f5dbc669a741252: Status 404 returned error can't find the container with id 928462429eb3d186aac0ab7c7e520e06e91f8b4fd7bb8faf1f5dbc669a741252
Sep 11 04:08:19.574534 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:19.574506    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:19.690429 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:19.690343    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:20.325693 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:20.325656    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589" event={"ID":"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94","Type":"ContainerStarted","Data":"fae16023c149f72cd4c0b215df6946381fad1d2efac9a01fd6235d1caa8a69fa"}
Sep 11 04:08:20.326147 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:20.325724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589" event={"ID":"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94","Type":"ContainerStarted","Data":"928462429eb3d186aac0ab7c7e520e06e91f8b4fd7bb8faf1f5dbc669a741252"}
Sep 11 04:08:20.326147 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:20.325961    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:20.345759 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:20.345705    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589" podStartSLOduration=1.8800352569999998 podStartE2EDuration="2.345690998s" totalImagesPullingTime="465.655741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:08:20.344109318 +0000 UTC m=+1092.966502373" watchObservedRunningTime="2026-09-11 04:08:20.345690998 +0000 UTC m=+1092.968084025"
Sep 11 04:08:24.627850 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:24.627806    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:26.336156 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:26.336128    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:08:34.042171 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:34.042132    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:34.143784 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:34.143748    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:42.546128 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:42.546093    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:08:54.111911 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:08:54.111870    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:09:11.835014 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:11.834966    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:09:36.405426 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:36.405329    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:09:37.407296 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.407265    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"]
Sep 11 04:09:37.407670 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.407470    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589" podUID="2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94" containerName="maas-api" containerID="cri-o://fae16023c149f72cd4c0b215df6946381fad1d2efac9a01fd6235d1caa8a69fa" gracePeriod=30
Sep 11 04:09:37.446461 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.446428    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:09:37.581319 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.581277    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:09:37.663787 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.663726    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:09:37.731419 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.731391    2581 generic.go:356] "Generic (PLEG): container finished" podID="2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94" containerID="fae16023c149f72cd4c0b215df6946381fad1d2efac9a01fd6235d1caa8a69fa" exitCode=0
Sep 11 04:09:37.731586 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.731436    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"
Sep 11 04:09:37.731586 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.731479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589" event={"ID":"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94","Type":"ContainerDied","Data":"fae16023c149f72cd4c0b215df6946381fad1d2efac9a01fd6235d1caa8a69fa"}
Sep 11 04:09:37.731586 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.731539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589" event={"ID":"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94","Type":"ContainerDied","Data":"928462429eb3d186aac0ab7c7e520e06e91f8b4fd7bb8faf1f5dbc669a741252"}
Sep 11 04:09:37.731586 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.731566    2581 scope.go:117] "RemoveContainer" containerID="fae16023c149f72cd4c0b215df6946381fad1d2efac9a01fd6235d1caa8a69fa"
Sep 11 04:09:37.741101 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.741085    2581 scope.go:117] "RemoveContainer" containerID="fae16023c149f72cd4c0b215df6946381fad1d2efac9a01fd6235d1caa8a69fa"
Sep 11 04:09:37.741403 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:09:37.741382    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae16023c149f72cd4c0b215df6946381fad1d2efac9a01fd6235d1caa8a69fa\": container with ID starting with fae16023c149f72cd4c0b215df6946381fad1d2efac9a01fd6235d1caa8a69fa not found: ID does not exist" containerID="fae16023c149f72cd4c0b215df6946381fad1d2efac9a01fd6235d1caa8a69fa"
Sep 11 04:09:37.741478 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.741411    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae16023c149f72cd4c0b215df6946381fad1d2efac9a01fd6235d1caa8a69fa"} err="failed to get container status \"fae16023c149f72cd4c0b215df6946381fad1d2efac9a01fd6235d1caa8a69fa\": rpc error: code = NotFound desc = could not find container \"fae16023c149f72cd4c0b215df6946381fad1d2efac9a01fd6235d1caa8a69fa\": container with ID starting with fae16023c149f72cd4c0b215df6946381fad1d2efac9a01fd6235d1caa8a69fa not found: ID does not exist"
Sep 11 04:09:37.789346 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.789324    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:09:37.842611 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.842589    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-metrics-certs\") pod \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") "
Sep 11 04:09:37.842758 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.842621    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-maas-api-tls\") pod \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") "
Sep 11 04:09:37.842758 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.842717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-openshift-service-ca\") pod \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") "
Sep 11 04:09:37.842758 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.842749    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d2sg\" (UniqueName: \"kubernetes.io/projected/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-kube-api-access-8d2sg\") pod \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\" (UID: \"2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94\") "
Sep 11 04:09:37.843378 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.843338    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94" (UID: "2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:09:37.845259 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.845232    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-kube-api-access-8d2sg" (OuterVolumeSpecName: "kube-api-access-8d2sg") pod "2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94" (UID: "2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94"). InnerVolumeSpecName "kube-api-access-8d2sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:09:37.845505 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.845463    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94" (UID: "2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:09:37.845603 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.845515    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94" (UID: "2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:09:37.943798 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.943734    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:09:37.943798 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.943760    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:09:37.943798 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.943769    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:09:37.943798 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.943777    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d2sg\" (UniqueName: \"kubernetes.io/projected/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94-kube-api-access-8d2sg\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:09:37.964992 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:37.964969    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:09:38.070004 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:38.069967    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"]
Sep 11 04:09:38.076987 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:38.076960    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0347a9fa-5bb84565c6-k7589"]
Sep 11 04:09:39.865254 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:39.865215    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94" path="/var/lib/kubelet/pods/2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94/volumes"
Sep 11 04:09:41.123045 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:41.122967    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:09:46.156458 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:09:46.156424    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:22.591863 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:22.591821    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:23.210496 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:23.210468    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"]
Sep 11 04:10:24.383344 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:24.383308    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:29.596483 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:29.596445    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:29.696649 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:29.696615    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:29.837934 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:29.837899    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:31.051985 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:31.051951    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:31.150685 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:31.150650    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:31.256455 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:31.256414    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:31.829373 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:31.829338    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:31.937203 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:31.937160    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:32.032404 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:32.032375    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:33.283069 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:33.283007    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:36.946517 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:36.946480    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:38.229431 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.229379    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn" podUID="c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d" containerName="main" containerID="cri-o://6b93e421a9a7f355323d68f76fcd75c96caa68cc22d4d1140a0f7d45fee2830f" gracePeriod=45
Sep 11 04:10:38.479290 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.479265    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:10:38.617765 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.617715    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-model-cache\") pod \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") "
Sep 11 04:10:38.617765 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.617775    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ldc7\" (UniqueName: \"kubernetes.io/projected/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-kube-api-access-6ldc7\") pod \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") "
Sep 11 04:10:38.618047 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.617798    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-dshm\") pod \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") "
Sep 11 04:10:38.618047 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.617863    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-tls-certs\") pod \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") "
Sep 11 04:10:38.618047 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.617899    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-home\") pod \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") "
Sep 11 04:10:38.618047 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.617940    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-tmp-dir\") pod \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\" (UID: \"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d\") "
Sep 11 04:10:38.618285 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.618075    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d" (UID: "c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:38.618285 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.618244    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-home" (OuterVolumeSpecName: "home") pod "c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d" (UID: "c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:38.618285 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.618254    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d" (UID: "c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:38.618415 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.618335    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-home\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:10:38.618415 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.618355    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-tmp-dir\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:10:38.618415 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.618372    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-model-cache\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:10:38.620107 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.620075    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d" (UID: "c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:10:38.620107 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.620088    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-dshm" (OuterVolumeSpecName: "dshm") pod "c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d" (UID: "c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:38.620254 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.620149    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-kube-api-access-6ldc7" (OuterVolumeSpecName: "kube-api-access-6ldc7") pod "c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d" (UID: "c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d"). InnerVolumeSpecName "kube-api-access-6ldc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:10:38.718909 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.718872    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-dshm\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:10:38.718909 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.718903    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-tls-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:10:38.718909 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:38.718917    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ldc7\" (UniqueName: \"kubernetes.io/projected/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d-kube-api-access-6ldc7\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:10:39.043264 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:39.043170    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:39.046315 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:39.046290    2581 generic.go:356] "Generic (PLEG): container finished" podID="c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d" containerID="6b93e421a9a7f355323d68f76fcd75c96caa68cc22d4d1140a0f7d45fee2830f" exitCode=0
Sep 11 04:10:39.046589 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:39.046343    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"
Sep 11 04:10:39.046589 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:39.046378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn" event={"ID":"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d","Type":"ContainerDied","Data":"6b93e421a9a7f355323d68f76fcd75c96caa68cc22d4d1140a0f7d45fee2830f"}
Sep 11 04:10:39.046589 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:39.046424    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn" event={"ID":"c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d","Type":"ContainerDied","Data":"56e74b33d1338dd72ab07e8af451e63039986673dea5331db5ef5bad8382f249"}
Sep 11 04:10:39.046589 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:39.046446    2581 scope.go:117] "RemoveContainer" containerID="6b93e421a9a7f355323d68f76fcd75c96caa68cc22d4d1140a0f7d45fee2830f"
Sep 11 04:10:39.055724 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:39.055705    2581 scope.go:117] "RemoveContainer" containerID="6b93e421a9a7f355323d68f76fcd75c96caa68cc22d4d1140a0f7d45fee2830f"
Sep 11 04:10:39.055991 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:10:39.055974    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b93e421a9a7f355323d68f76fcd75c96caa68cc22d4d1140a0f7d45fee2830f\": container with ID starting with 6b93e421a9a7f355323d68f76fcd75c96caa68cc22d4d1140a0f7d45fee2830f not found: ID does not exist" containerID="6b93e421a9a7f355323d68f76fcd75c96caa68cc22d4d1140a0f7d45fee2830f"
Sep 11 04:10:39.056060 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:39.056000    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b93e421a9a7f355323d68f76fcd75c96caa68cc22d4d1140a0f7d45fee2830f"} err="failed to get container status \"6b93e421a9a7f355323d68f76fcd75c96caa68cc22d4d1140a0f7d45fee2830f\": rpc error: code = NotFound desc = could not find container \"6b93e421a9a7f355323d68f76fcd75c96caa68cc22d4d1140a0f7d45fee2830f\": container with ID starting with 6b93e421a9a7f355323d68f76fcd75c96caa68cc22d4d1140a0f7d45fee2830f not found: ID does not exist"
Sep 11 04:10:39.067953 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:39.067932    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"]
Sep 11 04:10:39.072083 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:39.072060    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/auth-test-model-9d031d-kserve-5b79d66d67-c64fn"]
Sep 11 04:10:39.107354 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:39.107327    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:39.688394 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:39.688342    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:39.862873 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:39.862841    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d" path="/var/lib/kubelet/pods/c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d/volumes"
Sep 11 04:10:45.451873 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:45.451806    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:45.515241 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:45.515208    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:45.595752 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:45.595722    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:56.124696 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:56.124657    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:10:56.343036 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:10:56.342991    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:02.903531 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.903449    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"]
Sep 11 04:11:02.903945 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.903904    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d" containerName="main"
Sep 11 04:11:02.903945 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.903917    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d" containerName="main"
Sep 11 04:11:02.903945 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.903934    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94" containerName="maas-api"
Sep 11 04:11:02.903945 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.903940    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94" containerName="maas-api"
Sep 11 04:11:02.904095 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.904001    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e28c26-3f03-4c29-a0f1-c752cf0a6d3d" containerName="main"
Sep 11 04:11:02.904095 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.904009    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e1362c9-f4d8-4a41-99eb-b6cec8bb8f94" containerName="maas-api"
Sep 11 04:11:02.907156 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.907135    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:02.909881 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.909854    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c158b2\"/\"openshift-service-ca.crt\""
Sep 11 04:11:02.910055 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.909959    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c158b2\"/\"sub-test-model-c158b2-kserve-self-signed-certs\""
Sep 11 04:11:02.910272 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.910252    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c158b2\"/\"default-dockercfg-ps8cb\""
Sep 11 04:11:02.910498 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.910479    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c158b2\"/\"kube-root-ca.crt\""
Sep 11 04:11:02.920256 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.920231    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"]
Sep 11 04:11:02.936447 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.936418    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:02.949635 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.949613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-dshm\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:02.949745 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.949661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-home\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:02.949745 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.949678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-model-cache\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:02.949826 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.949750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d5j2\" (UniqueName: \"kubernetes.io/projected/cc8c9585-9217-4324-87df-6585c18cf977-kube-api-access-7d5j2\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:02.949870 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.949824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-tmp-dir\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:02.949870 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:02.949852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc8c9585-9217-4324-87df-6585c18cf977-tls-certs\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:03.050621 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:03.050589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-tmp-dir\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:03.050814 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:03.050634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc8c9585-9217-4324-87df-6585c18cf977-tls-certs\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:03.050814 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:03.050692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-dshm\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:03.050814 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:03.050752    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-home\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:03.050814 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:03.050776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-model-cache\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:03.050814 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:03.050801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d5j2\" (UniqueName: \"kubernetes.io/projected/cc8c9585-9217-4324-87df-6585c18cf977-kube-api-access-7d5j2\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:03.051111 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:03.051069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-tmp-dir\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:03.051184 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:03.051124    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-home\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:03.051235 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:03.051183    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-model-cache\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:03.052864 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:03.052842    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-dshm\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:03.053168 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:03.053149    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc8c9585-9217-4324-87df-6585c18cf977-tls-certs\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:03.058874 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:03.058855    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d5j2\" (UniqueName: \"kubernetes.io/projected/cc8c9585-9217-4324-87df-6585c18cf977-kube-api-access-7d5j2\") pod \"sub-test-model-c158b2-kserve-65c666948f-hccqh\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:03.216168 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:03.216081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:03.485816 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:03.485792    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"]
Sep 11 04:11:03.487365 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:11:03.487337    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc8c9585_9217_4324_87df_6585c18cf977.slice/crio-463c2116827ed2b4f8aa288c40257d1bab2ccf6d6c91b86e470bd7cf04205c10 WatchSource:0}: Error finding container 463c2116827ed2b4f8aa288c40257d1bab2ccf6d6c91b86e470bd7cf04205c10: Status 404 returned error can't find the container with id 463c2116827ed2b4f8aa288c40257d1bab2ccf6d6c91b86e470bd7cf04205c10
Sep 11 04:11:04.185542 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:04.185505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh" event={"ID":"cc8c9585-9217-4324-87df-6585c18cf977","Type":"ContainerStarted","Data":"9a7bdbe4bc5a8f11405355dfac2b7eff3d535692b74a8fcb11f1749ee5daf6bf"}
Sep 11 04:11:04.185903 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:04.185552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh" event={"ID":"cc8c9585-9217-4324-87df-6585c18cf977","Type":"ContainerStarted","Data":"463c2116827ed2b4f8aa288c40257d1bab2ccf6d6c91b86e470bd7cf04205c10"}
Sep 11 04:11:04.203390 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:04.203338    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh" podStartSLOduration=2.203323641 podStartE2EDuration="2.203323641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:11:04.201360571 +0000 UTC m=+1256.823753619" watchObservedRunningTime="2026-09-11 04:11:04.203323641 +0000 UTC m=+1256.825716726"
Sep 11 04:11:13.216671 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:13.216637    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:13.217183 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:13.216689    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:13.229570 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:13.229548    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:13.242575 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:13.242553    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:11:17.287452 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:17.287420    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:23.197388 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:23.197354    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:23.295461 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:23.295420    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:23.540642 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:23.540558    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:23.856441 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:23.856406    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:24.102775 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:24.102734    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:26.870664 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:26.870621    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:26.956295 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:26.956262    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:28.029887 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.029854    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"]
Sep 11 04:11:28.034790 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.034769    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:28.037342 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.037321    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a7eaac-serving-cert\""
Sep 11 04:11:28.049233 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.049209    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"]
Sep 11 04:11:28.081034 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.081003    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:28.177396 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.177359    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:28.205207 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.205178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aefdd4f9-7ad1-476e-815f-928ba61931ef-metrics-certs\") pod \"maas-api-e2e-api-a-a7eaac-db95c4474-l565d\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:28.205365 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.205242    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aefdd4f9-7ad1-476e-815f-928ba61931ef-openshift-service-ca\") pod \"maas-api-e2e-api-a-a7eaac-db95c4474-l565d\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:28.205365 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.205286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aefdd4f9-7ad1-476e-815f-928ba61931ef-maas-api-tls\") pod \"maas-api-e2e-api-a-a7eaac-db95c4474-l565d\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:28.205365 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.205325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5br2\" (UniqueName: \"kubernetes.io/projected/aefdd4f9-7ad1-476e-815f-928ba61931ef-kube-api-access-c5br2\") pod \"maas-api-e2e-api-a-a7eaac-db95c4474-l565d\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:28.276745 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.276710    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:28.306340 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.306315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aefdd4f9-7ad1-476e-815f-928ba61931ef-metrics-certs\") pod \"maas-api-e2e-api-a-a7eaac-db95c4474-l565d\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:28.306496 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.306371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aefdd4f9-7ad1-476e-815f-928ba61931ef-openshift-service-ca\") pod \"maas-api-e2e-api-a-a7eaac-db95c4474-l565d\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:28.306496 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.306414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aefdd4f9-7ad1-476e-815f-928ba61931ef-maas-api-tls\") pod \"maas-api-e2e-api-a-a7eaac-db95c4474-l565d\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:28.306496 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.306470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5br2\" (UniqueName: \"kubernetes.io/projected/aefdd4f9-7ad1-476e-815f-928ba61931ef-kube-api-access-c5br2\") pod \"maas-api-e2e-api-a-a7eaac-db95c4474-l565d\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:28.307092 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.307070    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aefdd4f9-7ad1-476e-815f-928ba61931ef-openshift-service-ca\") pod \"maas-api-e2e-api-a-a7eaac-db95c4474-l565d\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:28.308860 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.308832    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aefdd4f9-7ad1-476e-815f-928ba61931ef-maas-api-tls\") pod \"maas-api-e2e-api-a-a7eaac-db95c4474-l565d\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:28.308938 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.308890    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aefdd4f9-7ad1-476e-815f-928ba61931ef-metrics-certs\") pod \"maas-api-e2e-api-a-a7eaac-db95c4474-l565d\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:28.314906 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.314882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5br2\" (UniqueName: \"kubernetes.io/projected/aefdd4f9-7ad1-476e-815f-928ba61931ef-kube-api-access-c5br2\") pod \"maas-api-e2e-api-a-a7eaac-db95c4474-l565d\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:28.347368 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.347339    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:28.627474 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.627376    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:28.628561 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.628536    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"]
Sep 11 04:11:28.630692 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:11:28.630668    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefdd4f9_7ad1_476e_815f_928ba61931ef.slice/crio-6aa2977fc436157620726d0a38bceb21c23446ba39482d91485f541cb93570f8 WatchSource:0}: Error finding container 6aa2977fc436157620726d0a38bceb21c23446ba39482d91485f541cb93570f8: Status 404 returned error can't find the container with id 6aa2977fc436157620726d0a38bceb21c23446ba39482d91485f541cb93570f8
Sep 11 04:11:28.632357 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.632335    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 04:11:28.717949 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:28.717900    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:29.312991 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:29.312954    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d" event={"ID":"aefdd4f9-7ad1-476e-815f-928ba61931ef","Type":"ContainerStarted","Data":"638c2ebddd42f0177dcc5d86450088e364919bdf96782de166eaef8f59a0f9ab"}
Sep 11 04:11:29.313351 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:29.313012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d" event={"ID":"aefdd4f9-7ad1-476e-815f-928ba61931ef","Type":"ContainerStarted","Data":"6aa2977fc436157620726d0a38bceb21c23446ba39482d91485f541cb93570f8"}
Sep 11 04:11:29.313351 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:29.313062    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:29.350710 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:29.350660    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d" podStartSLOduration=0.781166628 podStartE2EDuration="1.350644954s" totalImagesPullingTime="569.478326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:11:29.347746313 +0000 UTC m=+1281.970139358" watchObservedRunningTime="2026-09-11 04:11:29.350644954 +0000 UTC m=+1281.973037982"
Sep 11 04:11:35.324656 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:35.324623    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:11:48.517647 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:48.517614    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:48.993655 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:48.993620    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:49.413160 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:49.413127    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"]
Sep 11 04:11:49.434942 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:49.434910    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:51.697772 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:51.697731    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:52.138379 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:52.138344    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:52.448131 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:52.447105    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:52.623141 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:52.623101    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:55.948805 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:55.948771    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:56.046034 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:56.045976    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:57.059744 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.059694    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"]
Sep 11 04:11:57.064163 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.064137    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:57.067044 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.067006    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4c6f83-serving-cert\""
Sep 11 04:11:57.076983 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.076949    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"]
Sep 11 04:11:57.100573 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.100543    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:57.180117 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.180081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-openshift-service-ca\") pod \"maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:57.180273 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.180133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c27m8\" (UniqueName: \"kubernetes.io/projected/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-kube-api-access-c27m8\") pod \"maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:57.180273 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.180195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-metrics-certs\") pod \"maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:57.180378 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.180289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-maas-api-tls\") pod \"maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:57.207971 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.207922    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:57.280727 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.280696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-openshift-service-ca\") pod \"maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:57.280905 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.280741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c27m8\" (UniqueName: \"kubernetes.io/projected/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-kube-api-access-c27m8\") pod \"maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:57.280905 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.280849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-metrics-certs\") pod \"maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:57.281064 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.280932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-maas-api-tls\") pod \"maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:57.281142 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:11:57.281084    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-serving-cert: secret "maas-api-e2e-api-b-4c6f83-serving-cert" not found
Sep 11 04:11:57.281203 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:11:57.281152    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-maas-api-tls podName:c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b nodeName:}" failed. No retries permitted until 2026-09-11 04:11:57.781132847 +0000 UTC m=+1310.403525861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-maas-api-tls") pod "maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc" (UID: "c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b") : secret "maas-api-e2e-api-b-4c6f83-serving-cert" not found
Sep 11 04:11:57.281455 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.281434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-openshift-service-ca\") pod \"maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:57.283626 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.283604    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-metrics-certs\") pod \"maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:57.288779 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.288757    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c27m8\" (UniqueName: \"kubernetes.io/projected/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-kube-api-access-c27m8\") pod \"maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:57.646423 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.646363    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:57.744060 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.744016    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:57.786559 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.786531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-maas-api-tls\") pod \"maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:57.788847 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.788825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-maas-api-tls\") pod \"maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:57.834917 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.834878    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:11:57.977269 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:57.977191    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:58.257955 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:58.257933    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"]
Sep 11 04:11:58.259367 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:11:58.259340    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc401c6e7_bf77_45ec_b074_f2d5c9f6dc7b.slice/crio-93db4c92d8f0c0edfe04d7c32e8974a144a9ba7368237a65f406d37645802efe WatchSource:0}: Error finding container 93db4c92d8f0c0edfe04d7c32e8974a144a9ba7368237a65f406d37645802efe: Status 404 returned error can't find the container with id 93db4c92d8f0c0edfe04d7c32e8974a144a9ba7368237a65f406d37645802efe
Sep 11 04:11:58.468149 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:58.468111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc" event={"ID":"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b","Type":"ContainerStarted","Data":"93db4c92d8f0c0edfe04d7c32e8974a144a9ba7368237a65f406d37645802efe"}
Sep 11 04:11:59.475633 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:59.475596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc" event={"ID":"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b","Type":"ContainerStarted","Data":"0092d897c4133a16e6fe6c127b0fe5d88a7a4472558ab0bcc4586697d6e9f844"}
Sep 11 04:11:59.476011 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:59.475699    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:11:59.498061 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:11:59.497990    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc" podStartSLOduration=2.13896977 podStartE2EDuration="2.497977529s" totalImagesPullingTime="359.007759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:11:59.496153112 +0000 UTC m=+1312.118546153" watchObservedRunningTime="2026-09-11 04:11:59.497977529 +0000 UTC m=+1312.120370555"
Sep 11 04:12:04.433133 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.433086    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh" podUID="cc8c9585-9217-4324-87df-6585c18cf977" containerName="main" containerID="cri-o://9a7bdbe4bc5a8f11405355dfac2b7eff3d535692b74a8fcb11f1749ee5daf6bf" gracePeriod=45
Sep 11 04:12:04.683990 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.683935    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:12:04.753305 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.753279    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-tmp-dir\") pod \"cc8c9585-9217-4324-87df-6585c18cf977\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") "
Sep 11 04:12:04.753450 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.753330    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc8c9585-9217-4324-87df-6585c18cf977-tls-certs\") pod \"cc8c9585-9217-4324-87df-6585c18cf977\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") "
Sep 11 04:12:04.753450 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.753377    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-home\") pod \"cc8c9585-9217-4324-87df-6585c18cf977\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") "
Sep 11 04:12:04.753450 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.753408    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d5j2\" (UniqueName: \"kubernetes.io/projected/cc8c9585-9217-4324-87df-6585c18cf977-kube-api-access-7d5j2\") pod \"cc8c9585-9217-4324-87df-6585c18cf977\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") "
Sep 11 04:12:04.753450 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.753429    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-model-cache\") pod \"cc8c9585-9217-4324-87df-6585c18cf977\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") "
Sep 11 04:12:04.753644 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.753506    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-dshm\") pod \"cc8c9585-9217-4324-87df-6585c18cf977\" (UID: \"cc8c9585-9217-4324-87df-6585c18cf977\") "
Sep 11 04:12:04.753644 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.753605    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc8c9585-9217-4324-87df-6585c18cf977" (UID: "cc8c9585-9217-4324-87df-6585c18cf977"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:12:04.753761 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.753634    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-home" (OuterVolumeSpecName: "home") pod "cc8c9585-9217-4324-87df-6585c18cf977" (UID: "cc8c9585-9217-4324-87df-6585c18cf977"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:12:04.753761 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.753712    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc8c9585-9217-4324-87df-6585c18cf977" (UID: "cc8c9585-9217-4324-87df-6585c18cf977"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:12:04.753951 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.753928    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-tmp-dir\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:12:04.753951 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.753949    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-home\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:12:04.754150 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.753958    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-model-cache\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:12:04.755593 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.755570    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8c9585-9217-4324-87df-6585c18cf977-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc8c9585-9217-4324-87df-6585c18cf977" (UID: "cc8c9585-9217-4324-87df-6585c18cf977"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:12:04.755678 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.755603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8c9585-9217-4324-87df-6585c18cf977-kube-api-access-7d5j2" (OuterVolumeSpecName: "kube-api-access-7d5j2") pod "cc8c9585-9217-4324-87df-6585c18cf977" (UID: "cc8c9585-9217-4324-87df-6585c18cf977"). InnerVolumeSpecName "kube-api-access-7d5j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:12:04.755678 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.755618    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-dshm" (OuterVolumeSpecName: "dshm") pod "cc8c9585-9217-4324-87df-6585c18cf977" (UID: "cc8c9585-9217-4324-87df-6585c18cf977"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:12:04.855472 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.855438    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc8c9585-9217-4324-87df-6585c18cf977-dshm\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:12:04.855472 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.855466    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc8c9585-9217-4324-87df-6585c18cf977-tls-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:12:04.855653 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:04.855487    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d5j2\" (UniqueName: \"kubernetes.io/projected/cc8c9585-9217-4324-87df-6585c18cf977-kube-api-access-7d5j2\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:12:05.485964 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:05.485937    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:12:05.510428 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:05.510393    2581 generic.go:356] "Generic (PLEG): container finished" podID="cc8c9585-9217-4324-87df-6585c18cf977" containerID="9a7bdbe4bc5a8f11405355dfac2b7eff3d535692b74a8fcb11f1749ee5daf6bf" exitCode=0
Sep 11 04:12:05.510599 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:05.510447    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"
Sep 11 04:12:05.510710 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:05.510688    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh" event={"ID":"cc8c9585-9217-4324-87df-6585c18cf977","Type":"ContainerDied","Data":"9a7bdbe4bc5a8f11405355dfac2b7eff3d535692b74a8fcb11f1749ee5daf6bf"}
Sep 11 04:12:05.510836 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:05.510817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh" event={"ID":"cc8c9585-9217-4324-87df-6585c18cf977","Type":"ContainerDied","Data":"463c2116827ed2b4f8aa288c40257d1bab2ccf6d6c91b86e470bd7cf04205c10"}
Sep 11 04:12:05.510940 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:05.510927    2581 scope.go:117] "RemoveContainer" containerID="9a7bdbe4bc5a8f11405355dfac2b7eff3d535692b74a8fcb11f1749ee5daf6bf"
Sep 11 04:12:05.521953 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:05.521935    2581 scope.go:117] "RemoveContainer" containerID="9a7bdbe4bc5a8f11405355dfac2b7eff3d535692b74a8fcb11f1749ee5daf6bf"
Sep 11 04:12:05.522588 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:12:05.522440    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a7bdbe4bc5a8f11405355dfac2b7eff3d535692b74a8fcb11f1749ee5daf6bf\": container with ID starting with 9a7bdbe4bc5a8f11405355dfac2b7eff3d535692b74a8fcb11f1749ee5daf6bf not found: ID does not exist" containerID="9a7bdbe4bc5a8f11405355dfac2b7eff3d535692b74a8fcb11f1749ee5daf6bf"
Sep 11 04:12:05.522588 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:05.522558    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7bdbe4bc5a8f11405355dfac2b7eff3d535692b74a8fcb11f1749ee5daf6bf"} err="failed to get container status \"9a7bdbe4bc5a8f11405355dfac2b7eff3d535692b74a8fcb11f1749ee5daf6bf\": rpc error: code = NotFound desc = could not find container \"9a7bdbe4bc5a8f11405355dfac2b7eff3d535692b74a8fcb11f1749ee5daf6bf\": container with ID starting with 9a7bdbe4bc5a8f11405355dfac2b7eff3d535692b74a8fcb11f1749ee5daf6bf not found: ID does not exist"
Sep 11 04:12:05.536353 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:05.536330    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"]
Sep 11 04:12:05.539467 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:05.539446    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/sub-test-model-c158b2-kserve-65c666948f-hccqh"]
Sep 11 04:12:05.862203 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:05.862168    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc8c9585-9217-4324-87df-6585c18cf977" path="/var/lib/kubelet/pods/cc8c9585-9217-4324-87df-6585c18cf977/volumes"
Sep 11 04:12:06.795451 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.795421    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"]
Sep 11 04:12:06.796100 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.796080    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8c9585-9217-4324-87df-6585c18cf977" containerName="main"
Sep 11 04:12:06.796100 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.796101    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8c9585-9217-4324-87df-6585c18cf977" containerName="main"
Sep 11 04:12:06.796233 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.796181    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc8c9585-9217-4324-87df-6585c18cf977" containerName="main"
Sep 11 04:12:06.800751 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.800730    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.804574 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.804551    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9d031d\"/\"rate-test-model-9d031d-kserve-self-signed-certs\""
Sep 11 04:12:06.804695 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.804648    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9d031d\"/\"kube-root-ca.crt\""
Sep 11 04:12:06.804787 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.804767    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9d031d\"/\"default-dockercfg-x45hg\""
Sep 11 04:12:06.804883 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.804865    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9d031d\"/\"openshift-service-ca.crt\""
Sep 11 04:12:06.808179 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.808157    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"]
Sep 11 04:12:06.831522 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.831495    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:06.873667 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.873642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-tmp-dir\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.873754 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.873685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-dshm\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.873754 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.873712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-model-cache\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.873826 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.873780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-home\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.873874 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.873861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vd9j\" (UniqueName: \"kubernetes.io/projected/5c07459c-7a86-4b6b-8878-902ed31cb730-kube-api-access-4vd9j\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.873911 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.873889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c07459c-7a86-4b6b-8878-902ed31cb730-tls-certs\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.975394 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.975360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-tmp-dir\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.975540 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.975423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-dshm\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.975540 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.975443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-model-cache\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.975540 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.975465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-home\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.975540 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.975525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vd9j\" (UniqueName: \"kubernetes.io/projected/5c07459c-7a86-4b6b-8878-902ed31cb730-kube-api-access-4vd9j\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.975771 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.975580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c07459c-7a86-4b6b-8878-902ed31cb730-tls-certs\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.975850 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.975770    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-tmp-dir\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.975850 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.975821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-model-cache\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.975948 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.975863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-home\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.977673 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.977655    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-dshm\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.977933 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.977917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c07459c-7a86-4b6b-8878-902ed31cb730-tls-certs\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:06.982956 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:06.982934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vd9j\" (UniqueName: \"kubernetes.io/projected/5c07459c-7a86-4b6b-8878-902ed31cb730-kube-api-access-4vd9j\") pod \"rate-test-model-9d031d-kserve-686964fc5f-6gq8k\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:07.112319 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:07.112288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:07.385863 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:07.385827    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"]
Sep 11 04:12:07.387143 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:12:07.387119    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c07459c_7a86_4b6b_8878_902ed31cb730.slice/crio-57e32c937435ef96fa0682c3739520eb74697b8542f04d48e3cd750954b45986 WatchSource:0}: Error finding container 57e32c937435ef96fa0682c3739520eb74697b8542f04d48e3cd750954b45986: Status 404 returned error can't find the container with id 57e32c937435ef96fa0682c3739520eb74697b8542f04d48e3cd750954b45986
Sep 11 04:12:07.531050 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:07.530997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k" event={"ID":"5c07459c-7a86-4b6b-8878-902ed31cb730","Type":"ContainerStarted","Data":"2969712429a92948ac530877bf6a07dd512a2da72940a147b87e6f4511d6168a"}
Sep 11 04:12:07.531231 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:07.531057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k" event={"ID":"5c07459c-7a86-4b6b-8878-902ed31cb730","Type":"ContainerStarted","Data":"57e32c937435ef96fa0682c3739520eb74697b8542f04d48e3cd750954b45986"}
Sep 11 04:12:07.547674 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:07.547629    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k" podStartSLOduration=1.547615322 podStartE2EDuration="1.547615322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:12:07.546798292 +0000 UTC m=+1320.169191351" watchObservedRunningTime="2026-09-11 04:12:07.547615322 +0000 UTC m=+1320.170008340"
Sep 11 04:12:09.391525 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:09.391488    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:17.112855 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:17.112810    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:17.112855 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:17.112864    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:17.125820 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:17.125795    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:17.598994 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:17.598964    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:12:20.878717 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:20.878678    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:23.880416 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:23.880373    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:24.645694 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:24.645657    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:25.530649 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:25.530603    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:35.153702 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:35.153623    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:38.512074 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.512039    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"]
Sep 11 04:12:38.515493 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.515470    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.524102 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.524074    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c158b2\"/\"default-dockercfg-ps8cb\""
Sep 11 04:12:38.524286 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.524261    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c158b2\"/\"rate-test-model-c158b2-kserve-self-signed-certs\""
Sep 11 04:12:38.524565 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.524536    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c158b2\"/\"kube-root-ca.crt\""
Sep 11 04:12:38.524740 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.524713    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c158b2\"/\"openshift-service-ca.crt\""
Sep 11 04:12:38.528226 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.528204    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"]
Sep 11 04:12:38.578289 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.578254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-home\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.578446 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.578303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-tmp-dir\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.578446 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.578327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-model-cache\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.578446 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.578418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-dshm\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.578588 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.578491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd5db\" (UniqueName: \"kubernetes.io/projected/151a9c7a-0f06-43fa-9153-1643e8d8ff82-kube-api-access-fd5db\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.578588 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.578545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151a9c7a-0f06-43fa-9153-1643e8d8ff82-tls-certs\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.609931 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.609895    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:38.679900 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.679863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151a9c7a-0f06-43fa-9153-1643e8d8ff82-tls-certs\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.680080 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.679951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-home\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.680080 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.679993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-tmp-dir\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.680168 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.680145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-model-cache\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.680212 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.680188    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-dshm\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.680260 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.680243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd5db\" (UniqueName: \"kubernetes.io/projected/151a9c7a-0f06-43fa-9153-1643e8d8ff82-kube-api-access-fd5db\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.680330 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.680313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-home\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.680405 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.680383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-tmp-dir\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.680511 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.680492    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-model-cache\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.682296 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.682263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-dshm\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.682427 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.682412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151a9c7a-0f06-43fa-9153-1643e8d8ff82-tls-certs\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.688284 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.688263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd5db\" (UniqueName: \"kubernetes.io/projected/151a9c7a-0f06-43fa-9153-1643e8d8ff82-kube-api-access-fd5db\") pod \"rate-test-model-c158b2-kserve-644d468f79-2xscr\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:38.833342 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:38.833312    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:39.108170 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.108146    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"]
Sep 11 04:12:39.109778 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:12:39.109748    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod151a9c7a_0f06_43fa_9153_1643e8d8ff82.slice/crio-f7f9e25d71f1fa15139b5f8f4c2892bea7a4483514ebedd2b5a5bd9e04ef1d9c WatchSource:0}: Error finding container f7f9e25d71f1fa15139b5f8f4c2892bea7a4483514ebedd2b5a5bd9e04ef1d9c: Status 404 returned error can't find the container with id f7f9e25d71f1fa15139b5f8f4c2892bea7a4483514ebedd2b5a5bd9e04ef1d9c
Sep 11 04:12:39.137660 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.137634    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"]
Sep 11 04:12:39.138344 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.138316    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d" podUID="aefdd4f9-7ad1-476e-815f-928ba61931ef" containerName="maas-api" containerID="cri-o://638c2ebddd42f0177dcc5d86450088e364919bdf96782de166eaef8f59a0f9ab" gracePeriod=30
Sep 11 04:12:39.168717 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.168683    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:39.269680 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.269601    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:39.368511 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.368398    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:39.383010 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.382982    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:12:39.471773 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.471742    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:39.488744 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.488720    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5br2\" (UniqueName: \"kubernetes.io/projected/aefdd4f9-7ad1-476e-815f-928ba61931ef-kube-api-access-c5br2\") pod \"aefdd4f9-7ad1-476e-815f-928ba61931ef\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") "
Sep 11 04:12:39.488909 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.488762    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aefdd4f9-7ad1-476e-815f-928ba61931ef-metrics-certs\") pod \"aefdd4f9-7ad1-476e-815f-928ba61931ef\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") "
Sep 11 04:12:39.488909 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.488800    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aefdd4f9-7ad1-476e-815f-928ba61931ef-openshift-service-ca\") pod \"aefdd4f9-7ad1-476e-815f-928ba61931ef\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") "
Sep 11 04:12:39.489086 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.488940    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aefdd4f9-7ad1-476e-815f-928ba61931ef-maas-api-tls\") pod \"aefdd4f9-7ad1-476e-815f-928ba61931ef\" (UID: \"aefdd4f9-7ad1-476e-815f-928ba61931ef\") "
Sep 11 04:12:39.489959 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.489929    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aefdd4f9-7ad1-476e-815f-928ba61931ef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aefdd4f9-7ad1-476e-815f-928ba61931ef" (UID: "aefdd4f9-7ad1-476e-815f-928ba61931ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:12:39.491501 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.491461    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aefdd4f9-7ad1-476e-815f-928ba61931ef-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aefdd4f9-7ad1-476e-815f-928ba61931ef" (UID: "aefdd4f9-7ad1-476e-815f-928ba61931ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:12:39.491695 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.491668    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefdd4f9-7ad1-476e-815f-928ba61931ef-kube-api-access-c5br2" (OuterVolumeSpecName: "kube-api-access-c5br2") pod "aefdd4f9-7ad1-476e-815f-928ba61931ef" (UID: "aefdd4f9-7ad1-476e-815f-928ba61931ef"). InnerVolumeSpecName "kube-api-access-c5br2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:12:39.491912 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.491890    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aefdd4f9-7ad1-476e-815f-928ba61931ef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aefdd4f9-7ad1-476e-815f-928ba61931ef" (UID: "aefdd4f9-7ad1-476e-815f-928ba61931ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:12:39.590818 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.590775    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5br2\" (UniqueName: \"kubernetes.io/projected/aefdd4f9-7ad1-476e-815f-928ba61931ef-kube-api-access-c5br2\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:12:39.590818 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.590811    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aefdd4f9-7ad1-476e-815f-928ba61931ef-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:12:39.590818 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.590820    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aefdd4f9-7ad1-476e-815f-928ba61931ef-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:12:39.590818 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.590828    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aefdd4f9-7ad1-476e-815f-928ba61931ef-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:12:39.701156 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.701126    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr" event={"ID":"151a9c7a-0f06-43fa-9153-1643e8d8ff82","Type":"ContainerStarted","Data":"c8a5f49f7688f9380aef34aa9c6de2eaab37d8ca518f738d937bc8037b9c87f6"}
Sep 11 04:12:39.701285 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.701167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr" event={"ID":"151a9c7a-0f06-43fa-9153-1643e8d8ff82","Type":"ContainerStarted","Data":"f7f9e25d71f1fa15139b5f8f4c2892bea7a4483514ebedd2b5a5bd9e04ef1d9c"}
Sep 11 04:12:39.702436 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.702415    2581 generic.go:356] "Generic (PLEG): container finished" podID="aefdd4f9-7ad1-476e-815f-928ba61931ef" containerID="638c2ebddd42f0177dcc5d86450088e364919bdf96782de166eaef8f59a0f9ab" exitCode=0
Sep 11 04:12:39.702521 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.702445    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"
Sep 11 04:12:39.702521 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.702455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d" event={"ID":"aefdd4f9-7ad1-476e-815f-928ba61931ef","Type":"ContainerDied","Data":"638c2ebddd42f0177dcc5d86450088e364919bdf96782de166eaef8f59a0f9ab"}
Sep 11 04:12:39.702521 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.702489    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d" event={"ID":"aefdd4f9-7ad1-476e-815f-928ba61931ef","Type":"ContainerDied","Data":"6aa2977fc436157620726d0a38bceb21c23446ba39482d91485f541cb93570f8"}
Sep 11 04:12:39.702521 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.702510    2581 scope.go:117] "RemoveContainer" containerID="638c2ebddd42f0177dcc5d86450088e364919bdf96782de166eaef8f59a0f9ab"
Sep 11 04:12:39.712066 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.712045    2581 scope.go:117] "RemoveContainer" containerID="638c2ebddd42f0177dcc5d86450088e364919bdf96782de166eaef8f59a0f9ab"
Sep 11 04:12:39.712327 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:12:39.712307    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638c2ebddd42f0177dcc5d86450088e364919bdf96782de166eaef8f59a0f9ab\": container with ID starting with 638c2ebddd42f0177dcc5d86450088e364919bdf96782de166eaef8f59a0f9ab not found: ID does not exist" containerID="638c2ebddd42f0177dcc5d86450088e364919bdf96782de166eaef8f59a0f9ab"
Sep 11 04:12:39.712379 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.712337    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638c2ebddd42f0177dcc5d86450088e364919bdf96782de166eaef8f59a0f9ab"} err="failed to get container status \"638c2ebddd42f0177dcc5d86450088e364919bdf96782de166eaef8f59a0f9ab\": rpc error: code = NotFound desc = could not find container \"638c2ebddd42f0177dcc5d86450088e364919bdf96782de166eaef8f59a0f9ab\": container with ID starting with 638c2ebddd42f0177dcc5d86450088e364919bdf96782de166eaef8f59a0f9ab not found: ID does not exist"
Sep 11 04:12:39.722207 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.722167    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr" podStartSLOduration=1.72215431 podStartE2EDuration="1.72215431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:12:39.719639897 +0000 UTC m=+1352.342032945" watchObservedRunningTime="2026-09-11 04:12:39.72215431 +0000 UTC m=+1352.344547337"
Sep 11 04:12:39.735843 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.735824    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"]
Sep 11 04:12:39.739500 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.739472    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a7eaac-db95c4474-l565d"]
Sep 11 04:12:39.757132 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.757107    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:39.862115 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:39.862085    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aefdd4f9-7ad1-476e-815f-928ba61931ef" path="/var/lib/kubelet/pods/aefdd4f9-7ad1-476e-815f-928ba61931ef/volumes"
Sep 11 04:12:45.976155 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:45.976114    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:46.074565 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:46.074532    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:46.224304 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:46.224258    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:46.658094 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:46.658055    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:48.833725 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:48.833688    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:48.834182 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:48.833742    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:48.846250 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:48.846229    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:49.769222 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:49.769189    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:12:53.243758 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:53.243726    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:12:56.706722 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:12:56.706685    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:02.412793 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.412754    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"]
Sep 11 04:13:02.413176 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.413081    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc" podUID="c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b" containerName="maas-api" containerID="cri-o://0092d897c4133a16e6fe6c127b0fe5d88a7a4472558ab0bcc4586697d6e9f844" gracePeriod=30
Sep 11 04:13:02.436522 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.436496    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:02.536229 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.536198    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:02.622845 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.622755    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:02.656769 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.656744    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:13:02.724141 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.724052    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c27m8\" (UniqueName: \"kubernetes.io/projected/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-kube-api-access-c27m8\") pod \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") "
Sep 11 04:13:02.724141 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.724124    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-maas-api-tls\") pod \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") "
Sep 11 04:13:02.724369 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.724191    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-metrics-certs\") pod \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") "
Sep 11 04:13:02.724369 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.724227    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-openshift-service-ca\") pod \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\" (UID: \"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b\") "
Sep 11 04:13:02.724705 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.724659    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b" (UID: "c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:13:02.726301 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.726270    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b" (UID: "c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:13:02.726463 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.726439    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b" (UID: "c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:13:02.726653 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.726638    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-kube-api-access-c27m8" (OuterVolumeSpecName: "kube-api-access-c27m8") pod "c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b" (UID: "c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b"). InnerVolumeSpecName "kube-api-access-c27m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:13:02.734751 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.734689    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:02.825451 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.825414    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c27m8\" (UniqueName: \"kubernetes.io/projected/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-kube-api-access-c27m8\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:02.825451 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.825441    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:02.825451 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.825450    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:02.825736 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.825461    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:02.830931 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.830903    2581 generic.go:356] "Generic (PLEG): container finished" podID="c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b" containerID="0092d897c4133a16e6fe6c127b0fe5d88a7a4472558ab0bcc4586697d6e9f844" exitCode=0
Sep 11 04:13:02.831110 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.830964    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"
Sep 11 04:13:02.831110 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.830994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc" event={"ID":"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b","Type":"ContainerDied","Data":"0092d897c4133a16e6fe6c127b0fe5d88a7a4472558ab0bcc4586697d6e9f844"}
Sep 11 04:13:02.831110 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.831092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc" event={"ID":"c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b","Type":"ContainerDied","Data":"93db4c92d8f0c0edfe04d7c32e8974a144a9ba7368237a65f406d37645802efe"}
Sep 11 04:13:02.831298 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.831144    2581 scope.go:117] "RemoveContainer" containerID="0092d897c4133a16e6fe6c127b0fe5d88a7a4472558ab0bcc4586697d6e9f844"
Sep 11 04:13:02.841375 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.841354    2581 scope.go:117] "RemoveContainer" containerID="0092d897c4133a16e6fe6c127b0fe5d88a7a4472558ab0bcc4586697d6e9f844"
Sep 11 04:13:02.841656 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:13:02.841632    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0092d897c4133a16e6fe6c127b0fe5d88a7a4472558ab0bcc4586697d6e9f844\": container with ID starting with 0092d897c4133a16e6fe6c127b0fe5d88a7a4472558ab0bcc4586697d6e9f844 not found: ID does not exist" containerID="0092d897c4133a16e6fe6c127b0fe5d88a7a4472558ab0bcc4586697d6e9f844"
Sep 11 04:13:02.841745 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.841668    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0092d897c4133a16e6fe6c127b0fe5d88a7a4472558ab0bcc4586697d6e9f844"} err="failed to get container status \"0092d897c4133a16e6fe6c127b0fe5d88a7a4472558ab0bcc4586697d6e9f844\": rpc error: code = NotFound desc = could not find container \"0092d897c4133a16e6fe6c127b0fe5d88a7a4472558ab0bcc4586697d6e9f844\": container with ID starting with 0092d897c4133a16e6fe6c127b0fe5d88a7a4472558ab0bcc4586697d6e9f844 not found: ID does not exist"
Sep 11 04:13:02.863030 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.863002    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"]
Sep 11 04:13:02.876163 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.876142    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4c6f83-86c9bfff7-jrhmc"]
Sep 11 04:13:02.975446 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:02.975368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:03.573558 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:03.573520    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:03.862853 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:03.862779    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b" path="/var/lib/kubelet/pods/c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b/volumes"
Sep 11 04:13:09.014869 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:09.014837    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:09.075824 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:09.075784    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:09.256924 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:09.256892    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:09.830477 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:09.830445    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:19.762193 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:19.762160    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"]
Sep 11 04:13:19.769826 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:19.769794    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:20.143920 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.143886    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"]
Sep 11 04:13:20.144188 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.144161    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd" podUID="9ebe43be-27d5-45d6-82ff-192bf0879aeb" containerName="maas-api" containerID="cri-o://9e185d1632a888d3a5f2531b3ca38a6490814627931e4c8e10f364274fbbf2ca" gracePeriod=30
Sep 11 04:13:20.171802 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.171770    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:20.272452 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.272397    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:20.361126 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.361091    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:20.386385 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.386358    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:13:20.431627 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.431519    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:20.504394 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.504364    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-maas-api-tls\") pod \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") "
Sep 11 04:13:20.504554 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.504415    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-metrics-certs\") pod \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") "
Sep 11 04:13:20.504554 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.504491    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebe43be-27d5-45d6-82ff-192bf0879aeb-openshift-service-ca\") pod \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") "
Sep 11 04:13:20.504686 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.504636    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbbpl\" (UniqueName: \"kubernetes.io/projected/9ebe43be-27d5-45d6-82ff-192bf0879aeb-kube-api-access-cbbpl\") pod \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\" (UID: \"9ebe43be-27d5-45d6-82ff-192bf0879aeb\") "
Sep 11 04:13:20.504867 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.504839    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ebe43be-27d5-45d6-82ff-192bf0879aeb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ebe43be-27d5-45d6-82ff-192bf0879aeb" (UID: "9ebe43be-27d5-45d6-82ff-192bf0879aeb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:13:20.504992 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.504971    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebe43be-27d5-45d6-82ff-192bf0879aeb-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:20.506553 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.506524    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9ebe43be-27d5-45d6-82ff-192bf0879aeb" (UID: "9ebe43be-27d5-45d6-82ff-192bf0879aeb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:13:20.506937 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.506917    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ebe43be-27d5-45d6-82ff-192bf0879aeb" (UID: "9ebe43be-27d5-45d6-82ff-192bf0879aeb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:13:20.507011 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.506919    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ebe43be-27d5-45d6-82ff-192bf0879aeb-kube-api-access-cbbpl" (OuterVolumeSpecName: "kube-api-access-cbbpl") pod "9ebe43be-27d5-45d6-82ff-192bf0879aeb" (UID: "9ebe43be-27d5-45d6-82ff-192bf0879aeb"). InnerVolumeSpecName "kube-api-access-cbbpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:13:20.605965 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.605926    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbbpl\" (UniqueName: \"kubernetes.io/projected/9ebe43be-27d5-45d6-82ff-192bf0879aeb-kube-api-access-cbbpl\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:20.605965 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.605959    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:20.605965 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.605972    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ebe43be-27d5-45d6-82ff-192bf0879aeb-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:20.702807 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.702727    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:20.923544 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.923510    2581 generic.go:356] "Generic (PLEG): container finished" podID="9ebe43be-27d5-45d6-82ff-192bf0879aeb" containerID="9e185d1632a888d3a5f2531b3ca38a6490814627931e4c8e10f364274fbbf2ca" exitCode=0
Sep 11 04:13:20.923993 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.923552    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"
Sep 11 04:13:20.923993 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.923590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd" event={"ID":"9ebe43be-27d5-45d6-82ff-192bf0879aeb","Type":"ContainerDied","Data":"9e185d1632a888d3a5f2531b3ca38a6490814627931e4c8e10f364274fbbf2ca"}
Sep 11 04:13:20.923993 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.923639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd" event={"ID":"9ebe43be-27d5-45d6-82ff-192bf0879aeb","Type":"ContainerDied","Data":"48b21957f556faa1d2260a7c17b4cc0b7a7e6257f264f3044e0c914fb5de4d36"}
Sep 11 04:13:20.923993 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.923660    2581 scope.go:117] "RemoveContainer" containerID="9e185d1632a888d3a5f2531b3ca38a6490814627931e4c8e10f364274fbbf2ca"
Sep 11 04:13:20.933500 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.933480    2581 scope.go:117] "RemoveContainer" containerID="9e185d1632a888d3a5f2531b3ca38a6490814627931e4c8e10f364274fbbf2ca"
Sep 11 04:13:20.933762 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:13:20.933743    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e185d1632a888d3a5f2531b3ca38a6490814627931e4c8e10f364274fbbf2ca\": container with ID starting with 9e185d1632a888d3a5f2531b3ca38a6490814627931e4c8e10f364274fbbf2ca not found: ID does not exist" containerID="9e185d1632a888d3a5f2531b3ca38a6490814627931e4c8e10f364274fbbf2ca"
Sep 11 04:13:20.933830 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.933775    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e185d1632a888d3a5f2531b3ca38a6490814627931e4c8e10f364274fbbf2ca"} err="failed to get container status \"9e185d1632a888d3a5f2531b3ca38a6490814627931e4c8e10f364274fbbf2ca\": rpc error: code = NotFound desc = could not find container \"9e185d1632a888d3a5f2531b3ca38a6490814627931e4c8e10f364274fbbf2ca\": container with ID starting with 9e185d1632a888d3a5f2531b3ca38a6490814627931e4c8e10f364274fbbf2ca not found: ID does not exist"
Sep 11 04:13:20.946274 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.946245    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"]
Sep 11 04:13:20.948229 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:20.948206    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-68ec26-dc64f6479-nszsd"]
Sep 11 04:13:21.862455 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:21.862425    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ebe43be-27d5-45d6-82ff-192bf0879aeb" path="/var/lib/kubelet/pods/9ebe43be-27d5-45d6-82ff-192bf0879aeb/volumes"
Sep 11 04:13:22.006724 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:22.006688    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"]
Sep 11 04:13:22.029501 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:22.029471    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:22.439610 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:22.439572    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:22.736774 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:22.736700    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:23.209849 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:23.209820    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:25.688844 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:25.688803    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:25.783833 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:25.783799    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:26.706675 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.706638    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"]
Sep 11 04:13:26.707267 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.707241    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ebe43be-27d5-45d6-82ff-192bf0879aeb" containerName="maas-api"
Sep 11 04:13:26.707267 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.707260    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ebe43be-27d5-45d6-82ff-192bf0879aeb" containerName="maas-api"
Sep 11 04:13:26.707415 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.707273    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aefdd4f9-7ad1-476e-815f-928ba61931ef" containerName="maas-api"
Sep 11 04:13:26.707415 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.707279    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aefdd4f9-7ad1-476e-815f-928ba61931ef" containerName="maas-api"
Sep 11 04:13:26.707415 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.707309    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b" containerName="maas-api"
Sep 11 04:13:26.707415 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.707318    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b" containerName="maas-api"
Sep 11 04:13:26.707631 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.707427    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c401c6e7-bf77-45ec-b074-f2d5c9f6dc7b" containerName="maas-api"
Sep 11 04:13:26.707631 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.707442    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aefdd4f9-7ad1-476e-815f-928ba61931ef" containerName="maas-api"
Sep 11 04:13:26.707631 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.707454    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ebe43be-27d5-45d6-82ff-192bf0879aeb" containerName="maas-api"
Sep 11 04:13:26.712087 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.712067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:26.714697 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.714679    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ef4b62-serving-cert\""
Sep 11 04:13:26.724977 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.724946    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"]
Sep 11 04:13:26.750144 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.750111    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:26.853355 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.853323    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:26.864064 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.864008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e286cdc9-5034-4491-a67d-630cbd73bb9e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:26.864197 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.864087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e286cdc9-5034-4491-a67d-630cbd73bb9e-metrics-certs\") pod \"maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:26.864250 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.864201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e286cdc9-5034-4491-a67d-630cbd73bb9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:26.864289 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.864248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lctbr\" (UniqueName: \"kubernetes.io/projected/e286cdc9-5034-4491-a67d-630cbd73bb9e-kube-api-access-lctbr\") pod \"maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:26.953258 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.953222    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:26.965785 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.965715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e286cdc9-5034-4491-a67d-630cbd73bb9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:26.965785 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.965760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lctbr\" (UniqueName: \"kubernetes.io/projected/e286cdc9-5034-4491-a67d-630cbd73bb9e-kube-api-access-lctbr\") pod \"maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:26.965928 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.965833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e286cdc9-5034-4491-a67d-630cbd73bb9e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:26.966061 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.966014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e286cdc9-5034-4491-a67d-630cbd73bb9e-metrics-certs\") pod \"maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:26.966468 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.966446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e286cdc9-5034-4491-a67d-630cbd73bb9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:26.968274 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.968253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e286cdc9-5034-4491-a67d-630cbd73bb9e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:26.968354 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.968302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e286cdc9-5034-4491-a67d-630cbd73bb9e-metrics-certs\") pod \"maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:26.974189 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:26.974166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lctbr\" (UniqueName: \"kubernetes.io/projected/e286cdc9-5034-4491-a67d-630cbd73bb9e-kube-api-access-lctbr\") pod \"maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:27.024210 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:27.024185    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:27.170050 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:27.170003    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:27.271859 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:27.271763    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:27.300457 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:27.300428    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"]
Sep 11 04:13:27.302491 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:13:27.302468    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode286cdc9_5034_4491_a67d_630cbd73bb9e.slice/crio-545d937f0d61b9962da52b6f99ce6a0383ff21c3e65a9f9e97eb580e7b97c960 WatchSource:0}: Error finding container 545d937f0d61b9962da52b6f99ce6a0383ff21c3e65a9f9e97eb580e7b97c960: Status 404 returned error can't find the container with id 545d937f0d61b9962da52b6f99ce6a0383ff21c3e65a9f9e97eb580e7b97c960
Sep 11 04:13:27.396732 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:27.396696    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:27.490481 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:27.490444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:27.964525 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:27.964486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x" event={"ID":"e286cdc9-5034-4491-a67d-630cbd73bb9e","Type":"ContainerStarted","Data":"6fe26d7a8bdc5b85806f72ac5b81f33c4139d64fca7e3ed2304bfa7f6e939117"}
Sep 11 04:13:27.964945 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:27.964546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x" event={"ID":"e286cdc9-5034-4491-a67d-630cbd73bb9e","Type":"ContainerStarted","Data":"545d937f0d61b9962da52b6f99ce6a0383ff21c3e65a9f9e97eb580e7b97c960"}
Sep 11 04:13:27.964945 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:27.964613    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:28.042229 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:28.042133    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x" podStartSLOduration=1.554129477 podStartE2EDuration="2.042116736s" totalImagesPullingTime="487.987259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:13:28.041654803 +0000 UTC m=+1400.664047857" watchObservedRunningTime="2026-09-11 04:13:28.042116736 +0000 UTC m=+1400.664509763"
Sep 11 04:13:28.755473 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:28.755432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:33.975229 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:33.975198    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:13:34.781730 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:34.781684    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k" podUID="5c07459c-7a86-4b6b-8878-902ed31cb730" containerName="main" containerID="cri-o://2969712429a92948ac530877bf6a07dd512a2da72940a147b87e6f4511d6168a" gracePeriod=45
Sep 11 04:13:35.003462 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.003426    2581 generic.go:356] "Generic (PLEG): container finished" podID="5c07459c-7a86-4b6b-8878-902ed31cb730" containerID="2969712429a92948ac530877bf6a07dd512a2da72940a147b87e6f4511d6168a" exitCode=0
Sep 11 04:13:35.003801 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.003471    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k" event={"ID":"5c07459c-7a86-4b6b-8878-902ed31cb730","Type":"ContainerDied","Data":"2969712429a92948ac530877bf6a07dd512a2da72940a147b87e6f4511d6168a"}
Sep 11 04:13:35.039894 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.039838    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:13:35.147427 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.147395    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-model-cache\") pod \"5c07459c-7a86-4b6b-8878-902ed31cb730\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") "
Sep 11 04:13:35.147607 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.147464    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vd9j\" (UniqueName: \"kubernetes.io/projected/5c07459c-7a86-4b6b-8878-902ed31cb730-kube-api-access-4vd9j\") pod \"5c07459c-7a86-4b6b-8878-902ed31cb730\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") "
Sep 11 04:13:35.147607 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.147514    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-tmp-dir\") pod \"5c07459c-7a86-4b6b-8878-902ed31cb730\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") "
Sep 11 04:13:35.147722 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.147615    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-dshm\") pod \"5c07459c-7a86-4b6b-8878-902ed31cb730\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") "
Sep 11 04:13:35.147722 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.147643    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c07459c-7a86-4b6b-8878-902ed31cb730" (UID: "5c07459c-7a86-4b6b-8878-902ed31cb730"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:35.147722 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.147667    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c07459c-7a86-4b6b-8878-902ed31cb730-tls-certs\") pod \"5c07459c-7a86-4b6b-8878-902ed31cb730\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") "
Sep 11 04:13:35.147722 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.147716    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-home\") pod \"5c07459c-7a86-4b6b-8878-902ed31cb730\" (UID: \"5c07459c-7a86-4b6b-8878-902ed31cb730\") "
Sep 11 04:13:35.147929 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.147743    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c07459c-7a86-4b6b-8878-902ed31cb730" (UID: "5c07459c-7a86-4b6b-8878-902ed31cb730"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:35.148056 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.148008    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-home" (OuterVolumeSpecName: "home") pod "5c07459c-7a86-4b6b-8878-902ed31cb730" (UID: "5c07459c-7a86-4b6b-8878-902ed31cb730"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:35.148130 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.148057    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-model-cache\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:35.148130 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.148073    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-tmp-dir\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:35.149936 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.149910    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c07459c-7a86-4b6b-8878-902ed31cb730-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c07459c-7a86-4b6b-8878-902ed31cb730" (UID: "5c07459c-7a86-4b6b-8878-902ed31cb730"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:13:35.150097 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.150081    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-dshm" (OuterVolumeSpecName: "dshm") pod "5c07459c-7a86-4b6b-8878-902ed31cb730" (UID: "5c07459c-7a86-4b6b-8878-902ed31cb730"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:35.150152 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.150122    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c07459c-7a86-4b6b-8878-902ed31cb730-kube-api-access-4vd9j" (OuterVolumeSpecName: "kube-api-access-4vd9j") pod "5c07459c-7a86-4b6b-8878-902ed31cb730" (UID: "5c07459c-7a86-4b6b-8878-902ed31cb730"). InnerVolumeSpecName "kube-api-access-4vd9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:13:35.249448 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.249418    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c07459c-7a86-4b6b-8878-902ed31cb730-tls-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:35.249448 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.249445    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-home\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:35.249448 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.249454    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vd9j\" (UniqueName: \"kubernetes.io/projected/5c07459c-7a86-4b6b-8878-902ed31cb730-kube-api-access-4vd9j\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:35.249662 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:35.249465    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c07459c-7a86-4b6b-8878-902ed31cb730-dshm\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:36.010219 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:36.010180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k" event={"ID":"5c07459c-7a86-4b6b-8878-902ed31cb730","Type":"ContainerDied","Data":"57e32c937435ef96fa0682c3739520eb74697b8542f04d48e3cd750954b45986"}
Sep 11 04:13:36.010633 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:36.010237    2581 scope.go:117] "RemoveContainer" containerID="2969712429a92948ac530877bf6a07dd512a2da72940a147b87e6f4511d6168a"
Sep 11 04:13:36.010633 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:36.010195    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"
Sep 11 04:13:36.028144 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:36.028121    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"]
Sep 11 04:13:36.032015 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:36.031991    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/rate-test-model-9d031d-kserve-686964fc5f-6gq8k"]
Sep 11 04:13:37.025427 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.025390    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr" podUID="151a9c7a-0f06-43fa-9153-1643e8d8ff82" containerName="main" containerID="cri-o://c8a5f49f7688f9380aef34aa9c6de2eaab37d8ca518f738d937bc8037b9c87f6" gracePeriod=45
Sep 11 04:13:37.271760 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.271738    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:13:37.370117 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.370088    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-dshm\") pod \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") "
Sep 11 04:13:37.370260 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.370133    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd5db\" (UniqueName: \"kubernetes.io/projected/151a9c7a-0f06-43fa-9153-1643e8d8ff82-kube-api-access-fd5db\") pod \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") "
Sep 11 04:13:37.370260 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.370181    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-tmp-dir\") pod \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") "
Sep 11 04:13:37.370337 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.370270    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151a9c7a-0f06-43fa-9153-1643e8d8ff82-tls-certs\") pod \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") "
Sep 11 04:13:37.370337 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.370290    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-home\") pod \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") "
Sep 11 04:13:37.370337 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.370330    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-model-cache\") pod \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\" (UID: \"151a9c7a-0f06-43fa-9153-1643e8d8ff82\") "
Sep 11 04:13:37.370495 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.370468    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "151a9c7a-0f06-43fa-9153-1643e8d8ff82" (UID: "151a9c7a-0f06-43fa-9153-1643e8d8ff82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:37.370590 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.370568    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-home" (OuterVolumeSpecName: "home") pod "151a9c7a-0f06-43fa-9153-1643e8d8ff82" (UID: "151a9c7a-0f06-43fa-9153-1643e8d8ff82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:37.370668 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.370643    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-tmp-dir\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:37.370719 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.370650    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-model-cache" (OuterVolumeSpecName: "model-cache") pod "151a9c7a-0f06-43fa-9153-1643e8d8ff82" (UID: "151a9c7a-0f06-43fa-9153-1643e8d8ff82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:37.372286 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.372266    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151a9c7a-0f06-43fa-9153-1643e8d8ff82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "151a9c7a-0f06-43fa-9153-1643e8d8ff82" (UID: "151a9c7a-0f06-43fa-9153-1643e8d8ff82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:13:37.372676 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.372647    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-dshm" (OuterVolumeSpecName: "dshm") pod "151a9c7a-0f06-43fa-9153-1643e8d8ff82" (UID: "151a9c7a-0f06-43fa-9153-1643e8d8ff82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:37.372676 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.372652    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151a9c7a-0f06-43fa-9153-1643e8d8ff82-kube-api-access-fd5db" (OuterVolumeSpecName: "kube-api-access-fd5db") pod "151a9c7a-0f06-43fa-9153-1643e8d8ff82" (UID: "151a9c7a-0f06-43fa-9153-1643e8d8ff82"). InnerVolumeSpecName "kube-api-access-fd5db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:13:37.471830 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.471799    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151a9c7a-0f06-43fa-9153-1643e8d8ff82-tls-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:37.471830 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.471823    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-home\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:37.471830 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.471833    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-model-cache\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:37.472063 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.471842    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151a9c7a-0f06-43fa-9153-1643e8d8ff82-dshm\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:37.472063 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.471850    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd5db\" (UniqueName: \"kubernetes.io/projected/151a9c7a-0f06-43fa-9153-1643e8d8ff82-kube-api-access-fd5db\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:37.862626 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:37.862592    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c07459c-7a86-4b6b-8878-902ed31cb730" path="/var/lib/kubelet/pods/5c07459c-7a86-4b6b-8878-902ed31cb730/volumes"
Sep 11 04:13:38.023400 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.023365    2581 generic.go:356] "Generic (PLEG): container finished" podID="151a9c7a-0f06-43fa-9153-1643e8d8ff82" containerID="c8a5f49f7688f9380aef34aa9c6de2eaab37d8ca518f738d937bc8037b9c87f6" exitCode=0
Sep 11 04:13:38.023604 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.023419    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"
Sep 11 04:13:38.023604 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.023451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr" event={"ID":"151a9c7a-0f06-43fa-9153-1643e8d8ff82","Type":"ContainerDied","Data":"c8a5f49f7688f9380aef34aa9c6de2eaab37d8ca518f738d937bc8037b9c87f6"}
Sep 11 04:13:38.023604 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.023505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr" event={"ID":"151a9c7a-0f06-43fa-9153-1643e8d8ff82","Type":"ContainerDied","Data":"f7f9e25d71f1fa15139b5f8f4c2892bea7a4483514ebedd2b5a5bd9e04ef1d9c"}
Sep 11 04:13:38.023604 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.023526    2581 scope.go:117] "RemoveContainer" containerID="c8a5f49f7688f9380aef34aa9c6de2eaab37d8ca518f738d937bc8037b9c87f6"
Sep 11 04:13:38.032865 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.032707    2581 scope.go:117] "RemoveContainer" containerID="c8a5f49f7688f9380aef34aa9c6de2eaab37d8ca518f738d937bc8037b9c87f6"
Sep 11 04:13:38.033083 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:13:38.032957    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a5f49f7688f9380aef34aa9c6de2eaab37d8ca518f738d937bc8037b9c87f6\": container with ID starting with c8a5f49f7688f9380aef34aa9c6de2eaab37d8ca518f738d937bc8037b9c87f6 not found: ID does not exist" containerID="c8a5f49f7688f9380aef34aa9c6de2eaab37d8ca518f738d937bc8037b9c87f6"
Sep 11 04:13:38.033083 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.032980    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a5f49f7688f9380aef34aa9c6de2eaab37d8ca518f738d937bc8037b9c87f6"} err="failed to get container status \"c8a5f49f7688f9380aef34aa9c6de2eaab37d8ca518f738d937bc8037b9c87f6\": rpc error: code = NotFound desc = could not find container \"c8a5f49f7688f9380aef34aa9c6de2eaab37d8ca518f738d937bc8037b9c87f6\": container with ID starting with c8a5f49f7688f9380aef34aa9c6de2eaab37d8ca518f738d937bc8037b9c87f6 not found: ID does not exist"
Sep 11 04:13:38.040780 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.040756    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"]
Sep 11 04:13:38.044895 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.044873    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/rate-test-model-c158b2-kserve-644d468f79-2xscr"]
Sep 11 04:13:38.258205 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.258121    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"]
Sep 11 04:13:38.258396 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.258372    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g" podUID="ba6220da-66a5-4eb4-9654-9fba8a4c84e3" containerName="maas-api" containerID="cri-o://5dc8e5217e1c03643703828897dbaa50a13a45f7c48055588fef148fed669800" gracePeriod=30
Sep 11 04:13:38.285839 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.285807    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:38.355338 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.355306    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:38.425722 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.425687    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:38.507346 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.507318    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:13:38.509082 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.508960    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:38.583017 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.582984    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f595j\" (UniqueName: \"kubernetes.io/projected/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-kube-api-access-f595j\") pod \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") "
Sep 11 04:13:38.583193 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.583044    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-maas-api-tls\") pod \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") "
Sep 11 04:13:38.583193 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.583131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-metrics-certs\") pod \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") "
Sep 11 04:13:38.583271 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.583244    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-openshift-service-ca\") pod \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\" (UID: \"ba6220da-66a5-4eb4-9654-9fba8a4c84e3\") "
Sep 11 04:13:38.583584 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.583560    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ba6220da-66a5-4eb4-9654-9fba8a4c84e3" (UID: "ba6220da-66a5-4eb4-9654-9fba8a4c84e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:13:38.585211 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.585191    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ba6220da-66a5-4eb4-9654-9fba8a4c84e3" (UID: "ba6220da-66a5-4eb4-9654-9fba8a4c84e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:13:38.585297 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.585225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ba6220da-66a5-4eb4-9654-9fba8a4c84e3" (UID: "ba6220da-66a5-4eb4-9654-9fba8a4c84e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:13:38.585297 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.585240    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-kube-api-access-f595j" (OuterVolumeSpecName: "kube-api-access-f595j") pod "ba6220da-66a5-4eb4-9654-9fba8a4c84e3" (UID: "ba6220da-66a5-4eb4-9654-9fba8a4c84e3"). InnerVolumeSpecName "kube-api-access-f595j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:13:38.684747 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.684715    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:38.684747 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.684744    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:38.684905 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.684757    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f595j\" (UniqueName: \"kubernetes.io/projected/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-kube-api-access-f595j\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:38.684905 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.684769    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba6220da-66a5-4eb4-9654-9fba8a4c84e3-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:13:38.837655 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:38.837624    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:39.032475 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:39.032445    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba6220da-66a5-4eb4-9654-9fba8a4c84e3" containerID="5dc8e5217e1c03643703828897dbaa50a13a45f7c48055588fef148fed669800" exitCode=0
Sep 11 04:13:39.032651 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:39.032489    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"
Sep 11 04:13:39.032651 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:39.032491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g" event={"ID":"ba6220da-66a5-4eb4-9654-9fba8a4c84e3","Type":"ContainerDied","Data":"5dc8e5217e1c03643703828897dbaa50a13a45f7c48055588fef148fed669800"}
Sep 11 04:13:39.032651 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:39.032596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g" event={"ID":"ba6220da-66a5-4eb4-9654-9fba8a4c84e3","Type":"ContainerDied","Data":"a62fad04bf6aeb0339013273d8823fe0eb127b97f7ab23b369c84354001712a4"}
Sep 11 04:13:39.032651 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:39.032620    2581 scope.go:117] "RemoveContainer" containerID="5dc8e5217e1c03643703828897dbaa50a13a45f7c48055588fef148fed669800"
Sep 11 04:13:39.042669 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:39.042650    2581 scope.go:117] "RemoveContainer" containerID="5dc8e5217e1c03643703828897dbaa50a13a45f7c48055588fef148fed669800"
Sep 11 04:13:39.042936 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:13:39.042921    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc8e5217e1c03643703828897dbaa50a13a45f7c48055588fef148fed669800\": container with ID starting with 5dc8e5217e1c03643703828897dbaa50a13a45f7c48055588fef148fed669800 not found: ID does not exist" containerID="5dc8e5217e1c03643703828897dbaa50a13a45f7c48055588fef148fed669800"
Sep 11 04:13:39.042979 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:39.042942    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc8e5217e1c03643703828897dbaa50a13a45f7c48055588fef148fed669800"} err="failed to get container status \"5dc8e5217e1c03643703828897dbaa50a13a45f7c48055588fef148fed669800\": rpc error: code = NotFound desc = could not find container \"5dc8e5217e1c03643703828897dbaa50a13a45f7c48055588fef148fed669800\": container with ID starting with 5dc8e5217e1c03643703828897dbaa50a13a45f7c48055588fef148fed669800 not found: ID does not exist"
Sep 11 04:13:39.056227 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:39.056200    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"]
Sep 11 04:13:39.058182 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:39.058155    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-604323-64cd984894-bq44g"]
Sep 11 04:13:39.862434 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:39.862403    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="151a9c7a-0f06-43fa-9153-1643e8d8ff82" path="/var/lib/kubelet/pods/151a9c7a-0f06-43fa-9153-1643e8d8ff82/volumes"
Sep 11 04:13:39.862812 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:39.862799    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba6220da-66a5-4eb4-9654-9fba8a4c84e3" path="/var/lib/kubelet/pods/ba6220da-66a5-4eb4-9654-9fba8a4c84e3/volumes"
Sep 11 04:13:44.740338 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:44.740302    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:44.803926 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:44.803895    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:45.007696 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:45.007548    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:47.940978 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:47.940942    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:50.414804 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:50.414759    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:51.039405 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:51.039363    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:51.433423 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:51.433385    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:51.941071 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:51.941010    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:54.513340 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:54.513278    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:54.602687 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:54.602653    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:55.616063 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.616014    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"]
Sep 11 04:13:55.616444 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.616430    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="151a9c7a-0f06-43fa-9153-1643e8d8ff82" containerName="main"
Sep 11 04:13:55.616444 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.616442    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="151a9c7a-0f06-43fa-9153-1643e8d8ff82" containerName="main"
Sep 11 04:13:55.616530 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.616453    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba6220da-66a5-4eb4-9654-9fba8a4c84e3" containerName="maas-api"
Sep 11 04:13:55.616530 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.616458    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba6220da-66a5-4eb4-9654-9fba8a4c84e3" containerName="maas-api"
Sep 11 04:13:55.616530 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.616478    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c07459c-7a86-4b6b-8878-902ed31cb730" containerName="main"
Sep 11 04:13:55.616530 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.616483    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c07459c-7a86-4b6b-8878-902ed31cb730" containerName="main"
Sep 11 04:13:55.616691 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.616547    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba6220da-66a5-4eb4-9654-9fba8a4c84e3" containerName="maas-api"
Sep 11 04:13:55.616691 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.616554    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="151a9c7a-0f06-43fa-9153-1643e8d8ff82" containerName="main"
Sep 11 04:13:55.616691 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.616561    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c07459c-7a86-4b6b-8878-902ed31cb730" containerName="main"
Sep 11 04:13:55.621418 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.621396    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:55.624693 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.624673    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-85dc29-serving-cert\""
Sep 11 04:13:55.635782 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.635756    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"]
Sep 11 04:13:55.659377 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.659343    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:55.716624 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.716598    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:55.750856 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.750829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31344e90-4d5c-4737-91fa-dadf53aca60b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:55.750982 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.750862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31344e90-4d5c-4737-91fa-dadf53aca60b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:55.750982 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.750919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzskc\" (UniqueName: \"kubernetes.io/projected/31344e90-4d5c-4737-91fa-dadf53aca60b-kube-api-access-rzskc\") pod \"maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:55.750982 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.750975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31344e90-4d5c-4737-91fa-dadf53aca60b-metrics-certs\") pod \"maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:55.817473 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.817441    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:55.851501 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.851470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31344e90-4d5c-4737-91fa-dadf53aca60b-metrics-certs\") pod \"maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:55.851603 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.851555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31344e90-4d5c-4737-91fa-dadf53aca60b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:55.851603 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.851580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31344e90-4d5c-4737-91fa-dadf53aca60b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:55.851677 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.851625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzskc\" (UniqueName: \"kubernetes.io/projected/31344e90-4d5c-4737-91fa-dadf53aca60b-kube-api-access-rzskc\") pod \"maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:55.852307 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.852285    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31344e90-4d5c-4737-91fa-dadf53aca60b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:55.853886 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.853866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31344e90-4d5c-4737-91fa-dadf53aca60b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:55.853959 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.853913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31344e90-4d5c-4737-91fa-dadf53aca60b-metrics-certs\") pod \"maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:55.862794 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.862773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzskc\" (UniqueName: \"kubernetes.io/projected/31344e90-4d5c-4737-91fa-dadf53aca60b-kube-api-access-rzskc\") pod \"maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:55.932091 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:55.932008    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:56.148234 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:56.148203    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:56.207431 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:56.207405    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"]
Sep 11 04:13:56.208522 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:13:56.208491    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31344e90_4d5c_4737_91fa_dadf53aca60b.slice/crio-9b58eede439d138a4fdbfb39c3d4b281a82b233af2ae0128f878404560926fc4 WatchSource:0}: Error finding container 9b58eede439d138a4fdbfb39c3d4b281a82b233af2ae0128f878404560926fc4: Status 404 returned error can't find the container with id 9b58eede439d138a4fdbfb39c3d4b281a82b233af2ae0128f878404560926fc4
Sep 11 04:13:56.247237 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:56.247208    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:13:57.125901 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:57.125869    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4" event={"ID":"31344e90-4d5c-4737-91fa-dadf53aca60b","Type":"ContainerStarted","Data":"289627a117b3642a091b350fa3387af433f62892a93ccddcd2638bee997414d1"}
Sep 11 04:13:57.126353 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:57.125918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4" event={"ID":"31344e90-4d5c-4737-91fa-dadf53aca60b","Type":"ContainerStarted","Data":"9b58eede439d138a4fdbfb39c3d4b281a82b233af2ae0128f878404560926fc4"}
Sep 11 04:13:57.126353 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:57.126074    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:13:57.147794 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:13:57.147745    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4" podStartSLOduration=1.806538813 podStartE2EDuration="2.14773173s" totalImagesPullingTime="341.192917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:13:57.145313141 +0000 UTC m=+1429.767706189" watchObservedRunningTime="2026-09-11 04:13:57.14773173 +0000 UTC m=+1429.770124757"
Sep 11 04:14:03.137390 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:14:03.137312    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:14:20.214802 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:14:20.214769    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:14:30.958162 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:14:30.958120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:14:31.017031 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:14:31.016992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:14:36.244531 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:14:36.244495    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:14:37.224050 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:14:37.223996    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:10.786955 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:10.786914    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:11.131254 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:11.131218    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:11.479798 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:11.479718    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:15.208525 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.208486    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:15.301893 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.301860    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:15.518900 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.518818    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"]
Sep 11 04:15:15.522910 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.522884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:15.526126 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.526103    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-27c12d-serving-cert\""
Sep 11 04:15:15.543713 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.543691    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"]
Sep 11 04:15:15.584777 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.584744    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:15.626898 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.626865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:15.627016 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.626904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:15.627016 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.626996    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:15.627126 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.627069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh87x\" (UniqueName: \"kubernetes.io/projected/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-kube-api-access-vh87x\") pod \"maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:15.690970 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.690933    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:15.728304 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.728268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:15.728304 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.728303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:15.728502 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.728343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:15.728502 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.728366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh87x\" (UniqueName: \"kubernetes.io/projected/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-kube-api-access-vh87x\") pod \"maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:15.728502 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:15:15.728480    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-serving-cert: secret "maas-api-e2e-ipp-cleanup-27c12d-serving-cert" not found
Sep 11 04:15:15.728652 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:15:15.728557    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-maas-api-tls podName:fa08c4cc-d220-4745-b617-2e6a74aa6cbe nodeName:}" failed. No retries permitted until 2026-09-11 04:15:16.228537942 +0000 UTC m=+1508.850930964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw" (UID: "fa08c4cc-d220-4745-b617-2e6a74aa6cbe") : secret "maas-api-e2e-ipp-cleanup-27c12d-serving-cert" not found
Sep 11 04:15:15.729069 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.729049    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:15.730850 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.730828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:15.739447 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:15.739423    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh87x\" (UniqueName: \"kubernetes.io/projected/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-kube-api-access-vh87x\") pod \"maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:16.069312 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:16.069277    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:16.175120 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:16.175082    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:16.234194 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:16.234156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:16.236488 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:16.236465    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:16.276605 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:16.276570    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:16.434719 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:16.434624    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:16.708822 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:16.708798    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"]
Sep 11 04:15:16.710939 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:15:16.710916    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa08c4cc_d220_4745_b617_2e6a74aa6cbe.slice/crio-cc71826b75b263f5fefc51296cc40f49e7fbc23d503d7263eaff201ee8f13fad WatchSource:0}: Error finding container cc71826b75b263f5fefc51296cc40f49e7fbc23d503d7263eaff201ee8f13fad: Status 404 returned error can't find the container with id cc71826b75b263f5fefc51296cc40f49e7fbc23d503d7263eaff201ee8f13fad
Sep 11 04:15:17.537357 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:17.537255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw" event={"ID":"fa08c4cc-d220-4745-b617-2e6a74aa6cbe","Type":"ContainerStarted","Data":"0fa4d1a883b4b620c12feff60d2c242aa1fdf111555699629a81043e2413fb66"}
Sep 11 04:15:17.537357 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:17.537309    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw" event={"ID":"fa08c4cc-d220-4745-b617-2e6a74aa6cbe","Type":"ContainerStarted","Data":"cc71826b75b263f5fefc51296cc40f49e7fbc23d503d7263eaff201ee8f13fad"}
Sep 11 04:15:17.537357 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:17.537333    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:17.571112 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:17.571061    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw" podStartSLOduration=2.035843772 podStartE2EDuration="2.571045941s" totalImagesPullingTime="535.202169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:15:17.567924928 +0000 UTC m=+1510.190317976" watchObservedRunningTime="2026-09-11 04:15:17.571045941 +0000 UTC m=+1510.193438968"
Sep 11 04:15:23.547972 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:23.547941    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:28.286823 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:28.286771    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:29.555840 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:29.555797    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:39.184877 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:39.184793    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:45.320900 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:45.320866    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:45.378774 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:45.378736    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:45.941646 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:45.941617    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"]
Sep 11 04:15:45.941880 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:45.941851    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw" podUID="fa08c4cc-d220-4745-b617-2e6a74aa6cbe" containerName="maas-api" containerID="cri-o://0fa4d1a883b4b620c12feff60d2c242aa1fdf111555699629a81043e2413fb66" gracePeriod=30
Sep 11 04:15:45.970790 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:45.970748    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:46.073123 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.073087    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:46.171103 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.171071    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:46.192738 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.192674    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:46.271406 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.271371    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:46.324184 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.324147    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-maas-api-tls\") pod \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") "
Sep 11 04:15:46.324547 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.324296    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh87x\" (UniqueName: \"kubernetes.io/projected/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-kube-api-access-vh87x\") pod \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") "
Sep 11 04:15:46.324547 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.324325    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-metrics-certs\") pod \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") "
Sep 11 04:15:46.324547 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.324374    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-openshift-service-ca\") pod \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\" (UID: \"fa08c4cc-d220-4745-b617-2e6a74aa6cbe\") "
Sep 11 04:15:46.324844 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.324811    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fa08c4cc-d220-4745-b617-2e6a74aa6cbe" (UID: "fa08c4cc-d220-4745-b617-2e6a74aa6cbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:15:46.326487 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.326460    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fa08c4cc-d220-4745-b617-2e6a74aa6cbe" (UID: "fa08c4cc-d220-4745-b617-2e6a74aa6cbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:15:46.326614 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.326488    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fa08c4cc-d220-4745-b617-2e6a74aa6cbe" (UID: "fa08c4cc-d220-4745-b617-2e6a74aa6cbe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:15:46.326614 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.326505    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-kube-api-access-vh87x" (OuterVolumeSpecName: "kube-api-access-vh87x") pod "fa08c4cc-d220-4745-b617-2e6a74aa6cbe" (UID: "fa08c4cc-d220-4745-b617-2e6a74aa6cbe"). InnerVolumeSpecName "kube-api-access-vh87x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:15:46.425430 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.425396    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:15:46.425430 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.425425    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh87x\" (UniqueName: \"kubernetes.io/projected/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-kube-api-access-vh87x\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:15:46.425430 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.425435    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:15:46.425663 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.425444    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa08c4cc-d220-4745-b617-2e6a74aa6cbe-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:15:46.565358 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.565319    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:46.685669 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.685639    2581 generic.go:356] "Generic (PLEG): container finished" podID="fa08c4cc-d220-4745-b617-2e6a74aa6cbe" containerID="0fa4d1a883b4b620c12feff60d2c242aa1fdf111555699629a81043e2413fb66" exitCode=0
Sep 11 04:15:46.685820 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.685702    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"
Sep 11 04:15:46.685820 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.685715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw" event={"ID":"fa08c4cc-d220-4745-b617-2e6a74aa6cbe","Type":"ContainerDied","Data":"0fa4d1a883b4b620c12feff60d2c242aa1fdf111555699629a81043e2413fb66"}
Sep 11 04:15:46.685820 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.685779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw" event={"ID":"fa08c4cc-d220-4745-b617-2e6a74aa6cbe","Type":"ContainerDied","Data":"cc71826b75b263f5fefc51296cc40f49e7fbc23d503d7263eaff201ee8f13fad"}
Sep 11 04:15:46.685820 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.685802    2581 scope.go:117] "RemoveContainer" containerID="0fa4d1a883b4b620c12feff60d2c242aa1fdf111555699629a81043e2413fb66"
Sep 11 04:15:46.694839 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.694822    2581 scope.go:117] "RemoveContainer" containerID="0fa4d1a883b4b620c12feff60d2c242aa1fdf111555699629a81043e2413fb66"
Sep 11 04:15:46.695145 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:15:46.695125    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa4d1a883b4b620c12feff60d2c242aa1fdf111555699629a81043e2413fb66\": container with ID starting with 0fa4d1a883b4b620c12feff60d2c242aa1fdf111555699629a81043e2413fb66 not found: ID does not exist" containerID="0fa4d1a883b4b620c12feff60d2c242aa1fdf111555699629a81043e2413fb66"
Sep 11 04:15:46.695202 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.695155    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa4d1a883b4b620c12feff60d2c242aa1fdf111555699629a81043e2413fb66"} err="failed to get container status \"0fa4d1a883b4b620c12feff60d2c242aa1fdf111555699629a81043e2413fb66\": rpc error: code = NotFound desc = could not find container \"0fa4d1a883b4b620c12feff60d2c242aa1fdf111555699629a81043e2413fb66\": container with ID starting with 0fa4d1a883b4b620c12feff60d2c242aa1fdf111555699629a81043e2413fb66 not found: ID does not exist"
Sep 11 04:15:46.706899 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.706873    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"]
Sep 11 04:15:46.709393 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:46.709374    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-27c12d-6c86d74dcf-p46xw"]
Sep 11 04:15:47.730298 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:47.730252    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:47.861459 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:47.861430    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa08c4cc-d220-4745-b617-2e6a74aa6cbe" path="/var/lib/kubelet/pods/fa08c4cc-d220-4745-b617-2e6a74aa6cbe/volumes"
Sep 11 04:15:52.500837 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:52.500800    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:52.570414 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:52.570375    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:52.832492 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:52.832453    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:15:56.425877 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:15:56.425843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:16:11.686204 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:11.686170    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"]
Sep 11 04:16:11.686849 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:11.686433    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x" podUID="e286cdc9-5034-4491-a67d-630cbd73bb9e" containerName="maas-api" containerID="cri-o://6fe26d7a8bdc5b85806f72ac5b81f33c4139d64fca7e3ed2304bfa7f6e939117" gracePeriod=30
Sep 11 04:16:11.719411 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:11.719373    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:16:11.788146 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:11.788112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:16:11.814746 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:11.814719    2581 generic.go:356] "Generic (PLEG): container finished" podID="e286cdc9-5034-4491-a67d-630cbd73bb9e" containerID="6fe26d7a8bdc5b85806f72ac5b81f33c4139d64fca7e3ed2304bfa7f6e939117" exitCode=0
Sep 11 04:16:11.814860 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:11.814793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x" event={"ID":"e286cdc9-5034-4491-a67d-630cbd73bb9e","Type":"ContainerDied","Data":"6fe26d7a8bdc5b85806f72ac5b81f33c4139d64fca7e3ed2304bfa7f6e939117"}
Sep 11 04:16:11.888994 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:11.888959    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:16:11.951448 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:11.951385    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:16:11.952322 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:11.952295    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:16:12.064247 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.064214    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e286cdc9-5034-4491-a67d-630cbd73bb9e-metrics-certs\") pod \"e286cdc9-5034-4491-a67d-630cbd73bb9e\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") "
Sep 11 04:16:12.064439 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.064280    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e286cdc9-5034-4491-a67d-630cbd73bb9e-openshift-service-ca\") pod \"e286cdc9-5034-4491-a67d-630cbd73bb9e\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") "
Sep 11 04:16:12.064439 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.064302    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lctbr\" (UniqueName: \"kubernetes.io/projected/e286cdc9-5034-4491-a67d-630cbd73bb9e-kube-api-access-lctbr\") pod \"e286cdc9-5034-4491-a67d-630cbd73bb9e\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") "
Sep 11 04:16:12.064439 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.064359    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e286cdc9-5034-4491-a67d-630cbd73bb9e-maas-api-tls\") pod \"e286cdc9-5034-4491-a67d-630cbd73bb9e\" (UID: \"e286cdc9-5034-4491-a67d-630cbd73bb9e\") "
Sep 11 04:16:12.064647 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.064620    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e286cdc9-5034-4491-a67d-630cbd73bb9e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e286cdc9-5034-4491-a67d-630cbd73bb9e" (UID: "e286cdc9-5034-4491-a67d-630cbd73bb9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:16:12.066438 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.066411    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e286cdc9-5034-4491-a67d-630cbd73bb9e-kube-api-access-lctbr" (OuterVolumeSpecName: "kube-api-access-lctbr") pod "e286cdc9-5034-4491-a67d-630cbd73bb9e" (UID: "e286cdc9-5034-4491-a67d-630cbd73bb9e"). InnerVolumeSpecName "kube-api-access-lctbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:16:12.066538 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.066437    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e286cdc9-5034-4491-a67d-630cbd73bb9e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e286cdc9-5034-4491-a67d-630cbd73bb9e" (UID: "e286cdc9-5034-4491-a67d-630cbd73bb9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:16:12.066538 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.066418    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e286cdc9-5034-4491-a67d-630cbd73bb9e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e286cdc9-5034-4491-a67d-630cbd73bb9e" (UID: "e286cdc9-5034-4491-a67d-630cbd73bb9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:16:12.165468 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.165437    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e286cdc9-5034-4491-a67d-630cbd73bb9e-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:16:12.165468 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.165463    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e286cdc9-5034-4491-a67d-630cbd73bb9e-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:16:12.165468 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.165474    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lctbr\" (UniqueName: \"kubernetes.io/projected/e286cdc9-5034-4491-a67d-630cbd73bb9e-kube-api-access-lctbr\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:16:12.165677 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.165484    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e286cdc9-5034-4491-a67d-630cbd73bb9e-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:16:12.294473 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.294351    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:16:12.821882 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.821851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x" event={"ID":"e286cdc9-5034-4491-a67d-630cbd73bb9e","Type":"ContainerDied","Data":"545d937f0d61b9962da52b6f99ce6a0383ff21c3e65a9f9e97eb580e7b97c960"}
Sep 11 04:16:12.822365 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.821914    2581 scope.go:117] "RemoveContainer" containerID="6fe26d7a8bdc5b85806f72ac5b81f33c4139d64fca7e3ed2304bfa7f6e939117"
Sep 11 04:16:12.822365 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.821861    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"
Sep 11 04:16:12.846519 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.846497    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"]
Sep 11 04:16:12.850862 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:12.850838    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ef4b62-6ddf58768d-7w88x"]
Sep 11 04:16:13.860823 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:13.860787    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e286cdc9-5034-4491-a67d-630cbd73bb9e" path="/var/lib/kubelet/pods/e286cdc9-5034-4491-a67d-630cbd73bb9e/volumes"
Sep 11 04:16:17.317760 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:17.317727    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:16:53.615758 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:53.615719    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:16:54.385362 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:54.385324    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:16:59.889547 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:16:59.889505    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:05.518654 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:05.518610    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:05.584846 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:05.584813    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:05.793737 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:05.793649    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:06.255868 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:06.255836    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:16.701928 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.701890    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"]
Sep 11 04:17:16.702728 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.702697    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4" podUID="31344e90-4d5c-4737-91fa-dadf53aca60b" containerName="maas-api" containerID="cri-o://289627a117b3642a091b350fa3387af433f62892a93ccddcd2638bee997414d1" gracePeriod=30
Sep 11 04:17:16.746570 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.746533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:16.825162 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.825108    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:16.903285 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.903226    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:16.955646 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.955592    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:17:16.979889 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.979856    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:16.982617 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.982597    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31344e90-4d5c-4737-91fa-dadf53aca60b-maas-api-tls\") pod \"31344e90-4d5c-4737-91fa-dadf53aca60b\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") "
Sep 11 04:17:16.982746 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.982640    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31344e90-4d5c-4737-91fa-dadf53aca60b-metrics-certs\") pod \"31344e90-4d5c-4737-91fa-dadf53aca60b\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") "
Sep 11 04:17:16.982746 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.982690    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31344e90-4d5c-4737-91fa-dadf53aca60b-openshift-service-ca\") pod \"31344e90-4d5c-4737-91fa-dadf53aca60b\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") "
Sep 11 04:17:16.982746 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.982733    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzskc\" (UniqueName: \"kubernetes.io/projected/31344e90-4d5c-4737-91fa-dadf53aca60b-kube-api-access-rzskc\") pod \"31344e90-4d5c-4737-91fa-dadf53aca60b\" (UID: \"31344e90-4d5c-4737-91fa-dadf53aca60b\") "
Sep 11 04:17:16.983587 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.983556    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31344e90-4d5c-4737-91fa-dadf53aca60b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "31344e90-4d5c-4737-91fa-dadf53aca60b" (UID: "31344e90-4d5c-4737-91fa-dadf53aca60b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:17:16.983809 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.983786    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31344e90-4d5c-4737-91fa-dadf53aca60b-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:17:16.984772 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.984748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31344e90-4d5c-4737-91fa-dadf53aca60b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "31344e90-4d5c-4737-91fa-dadf53aca60b" (UID: "31344e90-4d5c-4737-91fa-dadf53aca60b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:17:16.984941 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.984914    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31344e90-4d5c-4737-91fa-dadf53aca60b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "31344e90-4d5c-4737-91fa-dadf53aca60b" (UID: "31344e90-4d5c-4737-91fa-dadf53aca60b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:17:16.985286 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:16.985265    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31344e90-4d5c-4737-91fa-dadf53aca60b-kube-api-access-rzskc" (OuterVolumeSpecName: "kube-api-access-rzskc") pod "31344e90-4d5c-4737-91fa-dadf53aca60b" (UID: "31344e90-4d5c-4737-91fa-dadf53aca60b"). InnerVolumeSpecName "kube-api-access-rzskc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:17:17.056838 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.056802    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:17.085145 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.085115    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31344e90-4d5c-4737-91fa-dadf53aca60b-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:17:17.085145 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.085144    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31344e90-4d5c-4737-91fa-dadf53aca60b-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:17:17.085316 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.085158    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzskc\" (UniqueName: \"kubernetes.io/projected/31344e90-4d5c-4737-91fa-dadf53aca60b-kube-api-access-rzskc\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:17:17.152604 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.152570    2581 generic.go:356] "Generic (PLEG): container finished" podID="31344e90-4d5c-4737-91fa-dadf53aca60b" containerID="289627a117b3642a091b350fa3387af433f62892a93ccddcd2638bee997414d1" exitCode=0
Sep 11 04:17:17.152744 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.152613    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"
Sep 11 04:17:17.152744 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.152660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4" event={"ID":"31344e90-4d5c-4737-91fa-dadf53aca60b","Type":"ContainerDied","Data":"289627a117b3642a091b350fa3387af433f62892a93ccddcd2638bee997414d1"}
Sep 11 04:17:17.152744 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.152735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4" event={"ID":"31344e90-4d5c-4737-91fa-dadf53aca60b","Type":"ContainerDied","Data":"9b58eede439d138a4fdbfb39c3d4b281a82b233af2ae0128f878404560926fc4"}
Sep 11 04:17:17.152866 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.152760    2581 scope.go:117] "RemoveContainer" containerID="289627a117b3642a091b350fa3387af433f62892a93ccddcd2638bee997414d1"
Sep 11 04:17:17.162380 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.162363    2581 scope.go:117] "RemoveContainer" containerID="289627a117b3642a091b350fa3387af433f62892a93ccddcd2638bee997414d1"
Sep 11 04:17:17.162624 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:17:17.162604    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"289627a117b3642a091b350fa3387af433f62892a93ccddcd2638bee997414d1\": container with ID starting with 289627a117b3642a091b350fa3387af433f62892a93ccddcd2638bee997414d1 not found: ID does not exist" containerID="289627a117b3642a091b350fa3387af433f62892a93ccddcd2638bee997414d1"
Sep 11 04:17:17.162695 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.162635    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"289627a117b3642a091b350fa3387af433f62892a93ccddcd2638bee997414d1"} err="failed to get container status \"289627a117b3642a091b350fa3387af433f62892a93ccddcd2638bee997414d1\": rpc error: code = NotFound desc = could not find container \"289627a117b3642a091b350fa3387af433f62892a93ccddcd2638bee997414d1\": container with ID starting with 289627a117b3642a091b350fa3387af433f62892a93ccddcd2638bee997414d1 not found: ID does not exist"
Sep 11 04:17:17.177585 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.177559    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"]
Sep 11 04:17:17.183039 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.183006    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85dc29-756dc64979-4srg4"]
Sep 11 04:17:17.366349 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.366312    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:17.862181 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:17.862146    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31344e90-4d5c-4737-91fa-dadf53aca60b" path="/var/lib/kubelet/pods/31344e90-4d5c-4737-91fa-dadf53aca60b/volumes"
Sep 11 04:17:23.474767 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:23.474731    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:23.573291 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:23.573253    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:23.877405 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:23.877367    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:23.992725 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:23.992690    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:24.246146 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:24.246050    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:34.550581 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.550548    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"]
Sep 11 04:17:34.551112 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.550757    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l" podUID="d1364e0a-a2f6-4c5a-8b2d-65d63b3db641" containerName="maas-api" containerID="cri-o://514cafea76a8557b153316a5c39958b03bdf9a7a0bb28b82ed1f76fee9c2c244" gracePeriod=30
Sep 11 04:17:34.585278 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.585245    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:34.656774 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.656722    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:34.721232 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.721195    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:34.798482 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.798462    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:17:34.832683 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.832646    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:34.851855 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.851827    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf4pb\" (UniqueName: \"kubernetes.io/projected/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-kube-api-access-vf4pb\") pod \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") "
Sep 11 04:17:34.851989 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.851869    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-openshift-service-ca\") pod \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") "
Sep 11 04:17:34.851989 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.851892    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-maas-api-tls\") pod \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") "
Sep 11 04:17:34.851989 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.851947    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-metrics-certs\") pod \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\" (UID: \"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641\") "
Sep 11 04:17:34.852448 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.852422    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d1364e0a-a2f6-4c5a-8b2d-65d63b3db641" (UID: "d1364e0a-a2f6-4c5a-8b2d-65d63b3db641"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:17:34.853986 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.853956    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d1364e0a-a2f6-4c5a-8b2d-65d63b3db641" (UID: "d1364e0a-a2f6-4c5a-8b2d-65d63b3db641"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:17:34.854092 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.853984    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d1364e0a-a2f6-4c5a-8b2d-65d63b3db641" (UID: "d1364e0a-a2f6-4c5a-8b2d-65d63b3db641"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:17:34.854092 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.854009    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-kube-api-access-vf4pb" (OuterVolumeSpecName: "kube-api-access-vf4pb") pod "d1364e0a-a2f6-4c5a-8b2d-65d63b3db641" (UID: "d1364e0a-a2f6-4c5a-8b2d-65d63b3db641"). InnerVolumeSpecName "kube-api-access-vf4pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:17:34.891215 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.891174    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:34.953474 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.953437    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf4pb\" (UniqueName: \"kubernetes.io/projected/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-kube-api-access-vf4pb\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:17:34.953474 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.953472    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:17:34.953668 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.953486    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:17:34.953668 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:34.953502    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:17:35.180414 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:35.180329    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:35.249657 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:35.249624    2581 generic.go:356] "Generic (PLEG): container finished" podID="d1364e0a-a2f6-4c5a-8b2d-65d63b3db641" containerID="514cafea76a8557b153316a5c39958b03bdf9a7a0bb28b82ed1f76fee9c2c244" exitCode=0
Sep 11 04:17:35.249805 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:35.249689    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"
Sep 11 04:17:35.249805 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:35.249686    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l" event={"ID":"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641","Type":"ContainerDied","Data":"514cafea76a8557b153316a5c39958b03bdf9a7a0bb28b82ed1f76fee9c2c244"}
Sep 11 04:17:35.249937 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:35.249816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l" event={"ID":"d1364e0a-a2f6-4c5a-8b2d-65d63b3db641","Type":"ContainerDied","Data":"7d1834434dc81a7a8e7ab87a72cc10f1f2b3f53acd4261c761916950a88c147a"}
Sep 11 04:17:35.249937 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:35.249848    2581 scope.go:117] "RemoveContainer" containerID="514cafea76a8557b153316a5c39958b03bdf9a7a0bb28b82ed1f76fee9c2c244"
Sep 11 04:17:35.260980 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:35.260948    2581 scope.go:117] "RemoveContainer" containerID="514cafea76a8557b153316a5c39958b03bdf9a7a0bb28b82ed1f76fee9c2c244"
Sep 11 04:17:35.261263 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:17:35.261244    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"514cafea76a8557b153316a5c39958b03bdf9a7a0bb28b82ed1f76fee9c2c244\": container with ID starting with 514cafea76a8557b153316a5c39958b03bdf9a7a0bb28b82ed1f76fee9c2c244 not found: ID does not exist" containerID="514cafea76a8557b153316a5c39958b03bdf9a7a0bb28b82ed1f76fee9c2c244"
Sep 11 04:17:35.261317 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:35.261272    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514cafea76a8557b153316a5c39958b03bdf9a7a0bb28b82ed1f76fee9c2c244"} err="failed to get container status \"514cafea76a8557b153316a5c39958b03bdf9a7a0bb28b82ed1f76fee9c2c244\": rpc error: code = NotFound desc = could not find container \"514cafea76a8557b153316a5c39958b03bdf9a7a0bb28b82ed1f76fee9c2c244\": container with ID starting with 514cafea76a8557b153316a5c39958b03bdf9a7a0bb28b82ed1f76fee9c2c244 not found: ID does not exist"
Sep 11 04:17:35.274701 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:35.274665    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"]
Sep 11 04:17:35.281799 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:35.281781    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9d031d-7c4784799c-q6s4l"]
Sep 11 04:17:35.861950 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:35.861917    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1364e0a-a2f6-4c5a-8b2d-65d63b3db641" path="/var/lib/kubelet/pods/d1364e0a-a2f6-4c5a-8b2d-65d63b3db641/volumes"
Sep 11 04:17:41.292170 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:41.292134    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:41.400861 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:41.400823    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:41.625348 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:41.625301    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:42.033511 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:42.033427    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:51.502620 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:51.502581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:51.602065 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:51.602002    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:52.460395 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.460361    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"]
Sep 11 04:17:52.460591 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.460566    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh" podUID="5ec77937-b130-40f0-865d-aa13509e09e2" containerName="maas-api" containerID="cri-o://88f3ec39e789c8233dc12c8eda4c0817a5a17a0525d9d8c10aac116192bfcf9e" gracePeriod=30
Sep 11 04:17:52.494408 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.494378    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:52.592134 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.592096    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:52.692005 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.691964    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:52.711957 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.711904    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:17:52.793619 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.793571    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:52.825704 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.825673    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec77937-b130-40f0-865d-aa13509e09e2-maas-api-tls\") pod \"5ec77937-b130-40f0-865d-aa13509e09e2\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") "
Sep 11 04:17:52.825856 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.825749    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhf2x\" (UniqueName: \"kubernetes.io/projected/5ec77937-b130-40f0-865d-aa13509e09e2-kube-api-access-mhf2x\") pod \"5ec77937-b130-40f0-865d-aa13509e09e2\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") "
Sep 11 04:17:52.825856 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.825782    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec77937-b130-40f0-865d-aa13509e09e2-openshift-service-ca\") pod \"5ec77937-b130-40f0-865d-aa13509e09e2\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") "
Sep 11 04:17:52.825976 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.825890    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ec77937-b130-40f0-865d-aa13509e09e2-metrics-certs\") pod \"5ec77937-b130-40f0-865d-aa13509e09e2\" (UID: \"5ec77937-b130-40f0-865d-aa13509e09e2\") "
Sep 11 04:17:52.826254 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.826218    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec77937-b130-40f0-865d-aa13509e09e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ec77937-b130-40f0-865d-aa13509e09e2" (UID: "5ec77937-b130-40f0-865d-aa13509e09e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:17:52.827966 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.827933    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec77937-b130-40f0-865d-aa13509e09e2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5ec77937-b130-40f0-865d-aa13509e09e2" (UID: "5ec77937-b130-40f0-865d-aa13509e09e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:17:52.827966 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.827950    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec77937-b130-40f0-865d-aa13509e09e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ec77937-b130-40f0-865d-aa13509e09e2" (UID: "5ec77937-b130-40f0-865d-aa13509e09e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:17:52.827966 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.827952    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec77937-b130-40f0-865d-aa13509e09e2-kube-api-access-mhf2x" (OuterVolumeSpecName: "kube-api-access-mhf2x") pod "5ec77937-b130-40f0-865d-aa13509e09e2" (UID: "5ec77937-b130-40f0-865d-aa13509e09e2"). InnerVolumeSpecName "kube-api-access-mhf2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:17:52.927390 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.927360    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ec77937-b130-40f0-865d-aa13509e09e2-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:17:52.927390 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.927385    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec77937-b130-40f0-865d-aa13509e09e2-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:17:52.927390 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.927394    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhf2x\" (UniqueName: \"kubernetes.io/projected/5ec77937-b130-40f0-865d-aa13509e09e2-kube-api-access-mhf2x\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:17:52.927633 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:52.927403    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec77937-b130-40f0-865d-aa13509e09e2-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:17:53.104369 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:53.104337    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:53.340807 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:53.340776    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ec77937-b130-40f0-865d-aa13509e09e2" containerID="88f3ec39e789c8233dc12c8eda4c0817a5a17a0525d9d8c10aac116192bfcf9e" exitCode=0
Sep 11 04:17:53.340972 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:53.340815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh" event={"ID":"5ec77937-b130-40f0-865d-aa13509e09e2","Type":"ContainerDied","Data":"88f3ec39e789c8233dc12c8eda4c0817a5a17a0525d9d8c10aac116192bfcf9e"}
Sep 11 04:17:53.340972 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:53.340817    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"
Sep 11 04:17:53.340972 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:53.340852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh" event={"ID":"5ec77937-b130-40f0-865d-aa13509e09e2","Type":"ContainerDied","Data":"46c268b0107acb4176ed8ffb213380cea1f84fa3738f26f10435602cf872cfaf"}
Sep 11 04:17:53.340972 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:53.340878    2581 scope.go:117] "RemoveContainer" containerID="88f3ec39e789c8233dc12c8eda4c0817a5a17a0525d9d8c10aac116192bfcf9e"
Sep 11 04:17:53.350388 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:53.350366    2581 scope.go:117] "RemoveContainer" containerID="88f3ec39e789c8233dc12c8eda4c0817a5a17a0525d9d8c10aac116192bfcf9e"
Sep 11 04:17:53.350652 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:17:53.350632    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f3ec39e789c8233dc12c8eda4c0817a5a17a0525d9d8c10aac116192bfcf9e\": container with ID starting with 88f3ec39e789c8233dc12c8eda4c0817a5a17a0525d9d8c10aac116192bfcf9e not found: ID does not exist" containerID="88f3ec39e789c8233dc12c8eda4c0817a5a17a0525d9d8c10aac116192bfcf9e"
Sep 11 04:17:53.350710 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:53.350662    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f3ec39e789c8233dc12c8eda4c0817a5a17a0525d9d8c10aac116192bfcf9e"} err="failed to get container status \"88f3ec39e789c8233dc12c8eda4c0817a5a17a0525d9d8c10aac116192bfcf9e\": rpc error: code = NotFound desc = could not find container \"88f3ec39e789c8233dc12c8eda4c0817a5a17a0525d9d8c10aac116192bfcf9e\": container with ID starting with 88f3ec39e789c8233dc12c8eda4c0817a5a17a0525d9d8c10aac116192bfcf9e not found: ID does not exist"
Sep 11 04:17:53.371693 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:53.371628    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"]
Sep 11 04:17:53.373297 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:53.373278    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c158b2-8f477b5f4-2zglh"]
Sep 11 04:17:53.861130 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:53.861092    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ec77937-b130-40f0-865d-aa13509e09e2" path="/var/lib/kubelet/pods/5ec77937-b130-40f0-865d-aa13509e09e2/volumes"
Sep 11 04:17:59.341557 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:59.341519    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:59.445805 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:59.445771    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:17:59.630163 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:17:59.630091    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:19:23.056125 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:19:23.056082    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:19:23.344495 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:19:23.344409    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:19:25.192442 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:19:25.192409    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:20:19.800631 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:19.800595    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:20:27.138468 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:27.138429    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"]
Sep 11 04:20:27.140377 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:27.138626    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f" podUID="2b9b9624-1cf9-4d38-bd62-0c445e98b087" containerName="manager" containerID="cri-o://d9806b12c1b785c5f89376df67234b7fce2c20c19c509651325dc61783b89a8e" gracePeriod=10
Sep 11 04:20:29.091364 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.091338    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"
Sep 11 04:20:29.119897 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.119865    2581 generic.go:356] "Generic (PLEG): container finished" podID="2b9b9624-1cf9-4d38-bd62-0c445e98b087" containerID="d9806b12c1b785c5f89376df67234b7fce2c20c19c509651325dc61783b89a8e" exitCode=0
Sep 11 04:20:29.120074 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.119903    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"
Sep 11 04:20:29.120074 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.119947    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f" event={"ID":"2b9b9624-1cf9-4d38-bd62-0c445e98b087","Type":"ContainerDied","Data":"d9806b12c1b785c5f89376df67234b7fce2c20c19c509651325dc61783b89a8e"}
Sep 11 04:20:29.120074 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.120004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f" event={"ID":"2b9b9624-1cf9-4d38-bd62-0c445e98b087","Type":"ContainerDied","Data":"825cfa23e49eaac4ed7d8653d45c08ec94df59ae09738d14f692b8aedbe96c7f"}
Sep 11 04:20:29.120074 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.120052    2581 scope.go:117] "RemoveContainer" containerID="d9806b12c1b785c5f89376df67234b7fce2c20c19c509651325dc61783b89a8e"
Sep 11 04:20:29.131448 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.131423    2581 scope.go:117] "RemoveContainer" containerID="d9806b12c1b785c5f89376df67234b7fce2c20c19c509651325dc61783b89a8e"
Sep 11 04:20:29.131785 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:20:29.131759    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9806b12c1b785c5f89376df67234b7fce2c20c19c509651325dc61783b89a8e\": container with ID starting with d9806b12c1b785c5f89376df67234b7fce2c20c19c509651325dc61783b89a8e not found: ID does not exist" containerID="d9806b12c1b785c5f89376df67234b7fce2c20c19c509651325dc61783b89a8e"
Sep 11 04:20:29.131902 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.131799    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9806b12c1b785c5f89376df67234b7fce2c20c19c509651325dc61783b89a8e"} err="failed to get container status \"d9806b12c1b785c5f89376df67234b7fce2c20c19c509651325dc61783b89a8e\": rpc error: code = NotFound desc = could not find container \"d9806b12c1b785c5f89376df67234b7fce2c20c19c509651325dc61783b89a8e\": container with ID starting with d9806b12c1b785c5f89376df67234b7fce2c20c19c509651325dc61783b89a8e not found: ID does not exist"
Sep 11 04:20:29.172438 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.172403    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69hn5\" (UniqueName: \"kubernetes.io/projected/2b9b9624-1cf9-4d38-bd62-0c445e98b087-kube-api-access-69hn5\") pod \"2b9b9624-1cf9-4d38-bd62-0c445e98b087\" (UID: \"2b9b9624-1cf9-4d38-bd62-0c445e98b087\") "
Sep 11 04:20:29.172567 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.172462    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b9b9624-1cf9-4d38-bd62-0c445e98b087-extensions-socket-volume\") pod \"2b9b9624-1cf9-4d38-bd62-0c445e98b087\" (UID: \"2b9b9624-1cf9-4d38-bd62-0c445e98b087\") "
Sep 11 04:20:29.172886 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.172823    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9b9624-1cf9-4d38-bd62-0c445e98b087-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2b9b9624-1cf9-4d38-bd62-0c445e98b087" (UID: "2b9b9624-1cf9-4d38-bd62-0c445e98b087"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:20:29.174484 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.174451    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9b9624-1cf9-4d38-bd62-0c445e98b087-kube-api-access-69hn5" (OuterVolumeSpecName: "kube-api-access-69hn5") pod "2b9b9624-1cf9-4d38-bd62-0c445e98b087" (UID: "2b9b9624-1cf9-4d38-bd62-0c445e98b087"). InnerVolumeSpecName "kube-api-access-69hn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:20:29.273341 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.273305    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69hn5\" (UniqueName: \"kubernetes.io/projected/2b9b9624-1cf9-4d38-bd62-0c445e98b087-kube-api-access-69hn5\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:20:29.273341 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.273335    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b9b9624-1cf9-4d38-bd62-0c445e98b087-extensions-socket-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:20:29.443098 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.443067    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"]
Sep 11 04:20:29.447304 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.447279    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fnh8f"]
Sep 11 04:20:29.861654 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:29.861625    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b9b9624-1cf9-4d38-bd62-0c445e98b087" path="/var/lib/kubelet/pods/2b9b9624-1cf9-4d38-bd62-0c445e98b087/volumes"
Sep 11 04:20:32.212297 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:20:32.212262    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:21:06.455798 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:06.455751    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:21:27.816304 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816271    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c"]
Sep 11 04:21:27.816694 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816661    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1364e0a-a2f6-4c5a-8b2d-65d63b3db641" containerName="maas-api"
Sep 11 04:21:27.816694 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816673    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1364e0a-a2f6-4c5a-8b2d-65d63b3db641" containerName="maas-api"
Sep 11 04:21:27.816694 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816690    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e286cdc9-5034-4491-a67d-630cbd73bb9e" containerName="maas-api"
Sep 11 04:21:27.816694 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816696    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e286cdc9-5034-4491-a67d-630cbd73bb9e" containerName="maas-api"
Sep 11 04:21:27.816828 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816707    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31344e90-4d5c-4737-91fa-dadf53aca60b" containerName="maas-api"
Sep 11 04:21:27.816828 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816713    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="31344e90-4d5c-4737-91fa-dadf53aca60b" containerName="maas-api"
Sep 11 04:21:27.816828 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816719    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa08c4cc-d220-4745-b617-2e6a74aa6cbe" containerName="maas-api"
Sep 11 04:21:27.816828 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816725    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa08c4cc-d220-4745-b617-2e6a74aa6cbe" containerName="maas-api"
Sep 11 04:21:27.816828 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816740    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b9b9624-1cf9-4d38-bd62-0c445e98b087" containerName="manager"
Sep 11 04:21:27.816828 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816745    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b9b9624-1cf9-4d38-bd62-0c445e98b087" containerName="manager"
Sep 11 04:21:27.816828 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816752    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec77937-b130-40f0-865d-aa13509e09e2" containerName="maas-api"
Sep 11 04:21:27.816828 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816756    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec77937-b130-40f0-865d-aa13509e09e2" containerName="maas-api"
Sep 11 04:21:27.816828 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816811    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e286cdc9-5034-4491-a67d-630cbd73bb9e" containerName="maas-api"
Sep 11 04:21:27.816828 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816818    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="31344e90-4d5c-4737-91fa-dadf53aca60b" containerName="maas-api"
Sep 11 04:21:27.816828 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816824    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ec77937-b130-40f0-865d-aa13509e09e2" containerName="maas-api"
Sep 11 04:21:27.816828 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816830    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b9b9624-1cf9-4d38-bd62-0c445e98b087" containerName="manager"
Sep 11 04:21:27.817202 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816837    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1364e0a-a2f6-4c5a-8b2d-65d63b3db641" containerName="maas-api"
Sep 11 04:21:27.817202 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.816846    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa08c4cc-d220-4745-b617-2e6a74aa6cbe" containerName="maas-api"
Sep 11 04:21:27.819643 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.819628    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c"
Sep 11 04:21:27.823498 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.823476    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-762n8\""
Sep 11 04:21:27.845507 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.845480    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c"]
Sep 11 04:21:27.907115 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.907088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d52ae8a5-24b5-41f7-b7ca-060c409e6098-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l727c\" (UID: \"d52ae8a5-24b5-41f7-b7ca-060c409e6098\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c"
Sep 11 04:21:27.907284 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:27.907195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q89hp\" (UniqueName: \"kubernetes.io/projected/d52ae8a5-24b5-41f7-b7ca-060c409e6098-kube-api-access-q89hp\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l727c\" (UID: \"d52ae8a5-24b5-41f7-b7ca-060c409e6098\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c"
Sep 11 04:21:28.008731 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:28.008695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d52ae8a5-24b5-41f7-b7ca-060c409e6098-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l727c\" (UID: \"d52ae8a5-24b5-41f7-b7ca-060c409e6098\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c"
Sep 11 04:21:28.008894 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:28.008760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q89hp\" (UniqueName: \"kubernetes.io/projected/d52ae8a5-24b5-41f7-b7ca-060c409e6098-kube-api-access-q89hp\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l727c\" (UID: \"d52ae8a5-24b5-41f7-b7ca-060c409e6098\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c"
Sep 11 04:21:28.009099 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:28.009078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d52ae8a5-24b5-41f7-b7ca-060c409e6098-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l727c\" (UID: \"d52ae8a5-24b5-41f7-b7ca-060c409e6098\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c"
Sep 11 04:21:28.018533 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:28.018504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q89hp\" (UniqueName: \"kubernetes.io/projected/d52ae8a5-24b5-41f7-b7ca-060c409e6098-kube-api-access-q89hp\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l727c\" (UID: \"d52ae8a5-24b5-41f7-b7ca-060c409e6098\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c"
Sep 11 04:21:28.131271 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:28.131193    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c"
Sep 11 04:21:28.411909 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:28.411885    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c"]
Sep 11 04:21:28.413891 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:21:28.413862    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52ae8a5_24b5_41f7_b7ca_060c409e6098.slice/crio-41a85fdaf9efee41db5db1bf3731e1048f7dea987b80cd9c75b2c2b5e40be7ed WatchSource:0}: Error finding container 41a85fdaf9efee41db5db1bf3731e1048f7dea987b80cd9c75b2c2b5e40be7ed: Status 404 returned error can't find the container with id 41a85fdaf9efee41db5db1bf3731e1048f7dea987b80cd9c75b2c2b5e40be7ed
Sep 11 04:21:29.412305 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:29.412271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c" event={"ID":"d52ae8a5-24b5-41f7-b7ca-060c409e6098","Type":"ContainerStarted","Data":"6aea47a2d5934071dfde6679576dced57a08b26ef78a7052a1aff8175d1e5829"}
Sep 11 04:21:29.412755 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:29.412333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c" event={"ID":"d52ae8a5-24b5-41f7-b7ca-060c409e6098","Type":"ContainerStarted","Data":"41a85fdaf9efee41db5db1bf3731e1048f7dea987b80cd9c75b2c2b5e40be7ed"}
Sep 11 04:21:29.412755 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:29.412365    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c"
Sep 11 04:21:29.440925 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:29.440873    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c" podStartSLOduration=2.440857033 podStartE2EDuration="2.440857033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:21:29.437676584 +0000 UTC m=+1882.060069622" watchObservedRunningTime="2026-09-11 04:21:29.440857033 +0000 UTC m=+1882.063250059"
Sep 11 04:21:40.420072 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:21:40.419976    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l727c"
Sep 11 04:22:11.928640 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:11.928593    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:22:22.296671 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.296634    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"]
Sep 11 04:22:22.300324 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.300306    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.304611 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.304588    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-5ea2fe22-kserve-self-signed-certs\""
Sep 11 04:22:22.318438 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.318418    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"]
Sep 11 04:22:22.342169 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.342138    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:22:22.420070 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.420046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-tmp-dir\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.420190 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.420124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf4c411-3a4a-4346-afbe-aef656476d50-tls-certs\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.420190 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.420152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-model-cache\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.420190 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.420177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-dshm\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.420298 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.420199    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vzwz\" (UniqueName: \"kubernetes.io/projected/fdf4c411-3a4a-4346-afbe-aef656476d50-kube-api-access-8vzwz\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.420298 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.420254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-home\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.521292 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.521263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf4c411-3a4a-4346-afbe-aef656476d50-tls-certs\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.521440 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.521312    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-model-cache\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.521440 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.521339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-dshm\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.521440 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.521366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vzwz\" (UniqueName: \"kubernetes.io/projected/fdf4c411-3a4a-4346-afbe-aef656476d50-kube-api-access-8vzwz\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.521440 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.521399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-home\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.521653 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.521575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-tmp-dir\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.522014 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.521983    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-tmp-dir\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.522157 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.521991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-model-cache\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.522157 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.522042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-home\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.523728 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.523706    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-dshm\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.523933 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.523917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf4c411-3a4a-4346-afbe-aef656476d50-tls-certs\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.529906 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.529882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vzwz\" (UniqueName: \"kubernetes.io/projected/fdf4c411-3a4a-4346-afbe-aef656476d50-kube-api-access-8vzwz\") pod \"e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") " pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.609967 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.609934    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:22.881532 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:22.881500    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"]
Sep 11 04:22:22.884087 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:22:22.884062    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdf4c411_3a4a_4346_afbe_aef656476d50.slice/crio-e4ce32f1a52106c8dd2e784d9f55bd1a337aaf68788f5c871b6c629e0e564664 WatchSource:0}: Error finding container e4ce32f1a52106c8dd2e784d9f55bd1a337aaf68788f5c871b6c629e0e564664: Status 404 returned error can't find the container with id e4ce32f1a52106c8dd2e784d9f55bd1a337aaf68788f5c871b6c629e0e564664
Sep 11 04:22:23.683111 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:23.683073    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq" event={"ID":"fdf4c411-3a4a-4346-afbe-aef656476d50","Type":"ContainerStarted","Data":"c56fa637fc53f981fa6d644461199e5069a61f42cb6f14fa1179631b1d137ea8"}
Sep 11 04:22:23.683487 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:23.683121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq" event={"ID":"fdf4c411-3a4a-4346-afbe-aef656476d50","Type":"ContainerStarted","Data":"e4ce32f1a52106c8dd2e784d9f55bd1a337aaf68788f5c871b6c629e0e564664"}
Sep 11 04:22:23.699210 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:23.699148    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq" podStartSLOduration=1.699129955 podStartE2EDuration="1.699129955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:22:23.698491918 +0000 UTC m=+1936.320884944" watchObservedRunningTime="2026-09-11 04:22:23.699129955 +0000 UTC m=+1936.321522984"
Sep 11 04:22:32.610556 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:32.610509    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:32.611081 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:32.610589    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:32.624261 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:32.624233    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:32.742484 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:32.742454    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:22:44.912287 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:44.912255    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:22:46.485729 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:46.485693    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"]
Sep 11 04:22:46.508204 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:22:46.508176    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:23:01.506286 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.506191    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq" podUID="fdf4c411-3a4a-4346-afbe-aef656476d50" containerName="main" containerID="cri-o://c56fa637fc53f981fa6d644461199e5069a61f42cb6f14fa1179631b1d137ea8" gracePeriod=45
Sep 11 04:23:01.753961 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.753935    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:23:01.876782 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.876752    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-tmp-dir\") pod \"fdf4c411-3a4a-4346-afbe-aef656476d50\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") "
Sep 11 04:23:01.876782 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.876785    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-model-cache\") pod \"fdf4c411-3a4a-4346-afbe-aef656476d50\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") "
Sep 11 04:23:01.876972 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.876808    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-dshm\") pod \"fdf4c411-3a4a-4346-afbe-aef656476d50\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") "
Sep 11 04:23:01.876972 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.876900    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-home\") pod \"fdf4c411-3a4a-4346-afbe-aef656476d50\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") "
Sep 11 04:23:01.876972 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.876945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vzwz\" (UniqueName: \"kubernetes.io/projected/fdf4c411-3a4a-4346-afbe-aef656476d50-kube-api-access-8vzwz\") pod \"fdf4c411-3a4a-4346-afbe-aef656476d50\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") "
Sep 11 04:23:01.877150 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.876972    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf4c411-3a4a-4346-afbe-aef656476d50-tls-certs\") pod \"fdf4c411-3a4a-4346-afbe-aef656476d50\" (UID: \"fdf4c411-3a4a-4346-afbe-aef656476d50\") "
Sep 11 04:23:01.877150 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.877039    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdf4c411-3a4a-4346-afbe-aef656476d50" (UID: "fdf4c411-3a4a-4346-afbe-aef656476d50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:23:01.877150 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.877061    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdf4c411-3a4a-4346-afbe-aef656476d50" (UID: "fdf4c411-3a4a-4346-afbe-aef656476d50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:23:01.877311 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.877167    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-home" (OuterVolumeSpecName: "home") pod "fdf4c411-3a4a-4346-afbe-aef656476d50" (UID: "fdf4c411-3a4a-4346-afbe-aef656476d50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:23:01.877360 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.877340    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-tmp-dir\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:23:01.877360 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.877354    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-model-cache\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:23:01.877441 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.877362    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-home\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:23:01.879196 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.879172    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf4c411-3a4a-4346-afbe-aef656476d50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdf4c411-3a4a-4346-afbe-aef656476d50" (UID: "fdf4c411-3a4a-4346-afbe-aef656476d50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:23:01.879447 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.879422    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-dshm" (OuterVolumeSpecName: "dshm") pod "fdf4c411-3a4a-4346-afbe-aef656476d50" (UID: "fdf4c411-3a4a-4346-afbe-aef656476d50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:23:01.879447 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.879441    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf4c411-3a4a-4346-afbe-aef656476d50-kube-api-access-8vzwz" (OuterVolumeSpecName: "kube-api-access-8vzwz") pod "fdf4c411-3a4a-4346-afbe-aef656476d50" (UID: "fdf4c411-3a4a-4346-afbe-aef656476d50"). InnerVolumeSpecName "kube-api-access-8vzwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:23:01.883804 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.883784    2581 generic.go:356] "Generic (PLEG): container finished" podID="fdf4c411-3a4a-4346-afbe-aef656476d50" containerID="c56fa637fc53f981fa6d644461199e5069a61f42cb6f14fa1179631b1d137ea8" exitCode=0
Sep 11 04:23:01.883891 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.883838    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"
Sep 11 04:23:01.883891 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.883848    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq" event={"ID":"fdf4c411-3a4a-4346-afbe-aef656476d50","Type":"ContainerDied","Data":"c56fa637fc53f981fa6d644461199e5069a61f42cb6f14fa1179631b1d137ea8"}
Sep 11 04:23:01.883891 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.883883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq" event={"ID":"fdf4c411-3a4a-4346-afbe-aef656476d50","Type":"ContainerDied","Data":"e4ce32f1a52106c8dd2e784d9f55bd1a337aaf68788f5c871b6c629e0e564664"}
Sep 11 04:23:01.884002 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.883903    2581 scope.go:117] "RemoveContainer" containerID="c56fa637fc53f981fa6d644461199e5069a61f42cb6f14fa1179631b1d137ea8"
Sep 11 04:23:01.901696 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.901677    2581 scope.go:117] "RemoveContainer" containerID="c56fa637fc53f981fa6d644461199e5069a61f42cb6f14fa1179631b1d137ea8"
Sep 11 04:23:01.901982 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:23:01.901961    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c56fa637fc53f981fa6d644461199e5069a61f42cb6f14fa1179631b1d137ea8\": container with ID starting with c56fa637fc53f981fa6d644461199e5069a61f42cb6f14fa1179631b1d137ea8 not found: ID does not exist" containerID="c56fa637fc53f981fa6d644461199e5069a61f42cb6f14fa1179631b1d137ea8"
Sep 11 04:23:01.902067 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.901989    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c56fa637fc53f981fa6d644461199e5069a61f42cb6f14fa1179631b1d137ea8"} err="failed to get container status \"c56fa637fc53f981fa6d644461199e5069a61f42cb6f14fa1179631b1d137ea8\": rpc error: code = NotFound desc = could not find container \"c56fa637fc53f981fa6d644461199e5069a61f42cb6f14fa1179631b1d137ea8\": container with ID starting with c56fa637fc53f981fa6d644461199e5069a61f42cb6f14fa1179631b1d137ea8 not found: ID does not exist"
Sep 11 04:23:01.912912 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.912888    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"]
Sep 11 04:23:01.922725 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.922702    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-5ea2fe22-kserve-5985bdb4bc-9gfkq"]
Sep 11 04:23:01.979289 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.979263    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vzwz\" (UniqueName: \"kubernetes.io/projected/fdf4c411-3a4a-4346-afbe-aef656476d50-kube-api-access-8vzwz\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:23:01.979289 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.979290    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf4c411-3a4a-4346-afbe-aef656476d50-tls-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:23:01.979452 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:01.979305    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf4c411-3a4a-4346-afbe-aef656476d50-dshm\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:23:03.867075 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:03.862494    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdf4c411-3a4a-4346-afbe-aef656476d50" path="/var/lib/kubelet/pods/fdf4c411-3a4a-4346-afbe-aef656476d50/volumes"
Sep 11 04:23:10.030859 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.030818    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"]
Sep 11 04:23:10.031418 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.031395    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf4c411-3a4a-4346-afbe-aef656476d50" containerName="main"
Sep 11 04:23:10.031504 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.031420    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf4c411-3a4a-4346-afbe-aef656476d50" containerName="main"
Sep 11 04:23:10.031575 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.031547    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf4c411-3a4a-4346-afbe-aef656476d50" containerName="main"
Sep 11 04:23:10.035989 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.035969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:10.039792 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.039765    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 04:23:10.039900 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.039774    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 04:23:10.039900 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.039774    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:23:10.039900 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.039841    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wb8bv\""
Sep 11 04:23:10.040078 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.039895    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 04:23:10.044726 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.044702    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"]
Sep 11 04:23:10.050775 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.050753    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"]
Sep 11 04:23:10.054218 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.054195    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:10.065095 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.065072    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"]
Sep 11 04:23:10.155321 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.155284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e19559-2350-40cb-8a17-b02cb4193830-metrics-certs\") pod \"maas-api-cfc75856f-sqnk8\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:10.155321 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.155324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgb5b\" (UniqueName: \"kubernetes.io/projected/6cc98a94-4d24-42b8-aaf2-a28557b3d5c1-kube-api-access-fgb5b\") pod \"maas-api-6c9d99c6fc-9kgms\" (UID: \"6cc98a94-4d24-42b8-aaf2-a28557b3d5c1\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:10.155533 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.155356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e19559-2350-40cb-8a17-b02cb4193830-openshift-service-ca\") pod \"maas-api-cfc75856f-sqnk8\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:10.155533 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.155382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc98a94-4d24-42b8-aaf2-a28557b3d5c1-maas-api-tls\") pod \"maas-api-6c9d99c6fc-9kgms\" (UID: \"6cc98a94-4d24-42b8-aaf2-a28557b3d5c1\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:10.155533 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.155404    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc98a94-4d24-42b8-aaf2-a28557b3d5c1-metrics-certs\") pod \"maas-api-6c9d99c6fc-9kgms\" (UID: \"6cc98a94-4d24-42b8-aaf2-a28557b3d5c1\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:10.155533 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.155427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e19559-2350-40cb-8a17-b02cb4193830-maas-api-tls\") pod \"maas-api-cfc75856f-sqnk8\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:10.155533 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.155501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc98a94-4d24-42b8-aaf2-a28557b3d5c1-openshift-service-ca\") pod \"maas-api-6c9d99c6fc-9kgms\" (UID: \"6cc98a94-4d24-42b8-aaf2-a28557b3d5c1\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:10.155533 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.155522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dczs2\" (UniqueName: \"kubernetes.io/projected/52e19559-2350-40cb-8a17-b02cb4193830-kube-api-access-dczs2\") pod \"maas-api-cfc75856f-sqnk8\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:10.256444 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.256403    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e19559-2350-40cb-8a17-b02cb4193830-metrics-certs\") pod \"maas-api-cfc75856f-sqnk8\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:10.256444 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.256448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgb5b\" (UniqueName: \"kubernetes.io/projected/6cc98a94-4d24-42b8-aaf2-a28557b3d5c1-kube-api-access-fgb5b\") pod \"maas-api-6c9d99c6fc-9kgms\" (UID: \"6cc98a94-4d24-42b8-aaf2-a28557b3d5c1\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:10.256666 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.256477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e19559-2350-40cb-8a17-b02cb4193830-openshift-service-ca\") pod \"maas-api-cfc75856f-sqnk8\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:10.256666 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.256500    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc98a94-4d24-42b8-aaf2-a28557b3d5c1-maas-api-tls\") pod \"maas-api-6c9d99c6fc-9kgms\" (UID: \"6cc98a94-4d24-42b8-aaf2-a28557b3d5c1\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:10.256666 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.256523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc98a94-4d24-42b8-aaf2-a28557b3d5c1-metrics-certs\") pod \"maas-api-6c9d99c6fc-9kgms\" (UID: \"6cc98a94-4d24-42b8-aaf2-a28557b3d5c1\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:10.256666 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.256551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e19559-2350-40cb-8a17-b02cb4193830-maas-api-tls\") pod \"maas-api-cfc75856f-sqnk8\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:10.256666 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.256630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc98a94-4d24-42b8-aaf2-a28557b3d5c1-openshift-service-ca\") pod \"maas-api-6c9d99c6fc-9kgms\" (UID: \"6cc98a94-4d24-42b8-aaf2-a28557b3d5c1\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:10.256666 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.256650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dczs2\" (UniqueName: \"kubernetes.io/projected/52e19559-2350-40cb-8a17-b02cb4193830-kube-api-access-dczs2\") pod \"maas-api-cfc75856f-sqnk8\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:10.257372 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.257342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e19559-2350-40cb-8a17-b02cb4193830-openshift-service-ca\") pod \"maas-api-cfc75856f-sqnk8\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:10.257518 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.257415    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc98a94-4d24-42b8-aaf2-a28557b3d5c1-openshift-service-ca\") pod \"maas-api-6c9d99c6fc-9kgms\" (UID: \"6cc98a94-4d24-42b8-aaf2-a28557b3d5c1\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:10.259143 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.259116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e19559-2350-40cb-8a17-b02cb4193830-maas-api-tls\") pod \"maas-api-cfc75856f-sqnk8\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:10.259246 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.259116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e19559-2350-40cb-8a17-b02cb4193830-metrics-certs\") pod \"maas-api-cfc75856f-sqnk8\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:10.259389 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.259373    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc98a94-4d24-42b8-aaf2-a28557b3d5c1-metrics-certs\") pod \"maas-api-6c9d99c6fc-9kgms\" (UID: \"6cc98a94-4d24-42b8-aaf2-a28557b3d5c1\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:10.259456 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.259441    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc98a94-4d24-42b8-aaf2-a28557b3d5c1-maas-api-tls\") pod \"maas-api-6c9d99c6fc-9kgms\" (UID: \"6cc98a94-4d24-42b8-aaf2-a28557b3d5c1\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:10.264489 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.264467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dczs2\" (UniqueName: \"kubernetes.io/projected/52e19559-2350-40cb-8a17-b02cb4193830-kube-api-access-dczs2\") pod \"maas-api-cfc75856f-sqnk8\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:10.264593 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.264542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgb5b\" (UniqueName: \"kubernetes.io/projected/6cc98a94-4d24-42b8-aaf2-a28557b3d5c1-kube-api-access-fgb5b\") pod \"maas-api-6c9d99c6fc-9kgms\" (UID: \"6cc98a94-4d24-42b8-aaf2-a28557b3d5c1\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:10.347857 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.347832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:10.365593 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.365562    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:10.623961 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.623883    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"]
Sep 11 04:23:10.627539 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:23:10.627504    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e19559_2350_40cb_8a17_b02cb4193830.slice/crio-dff2fc763f3c28be5127113adbacdd95f34f94f19ca2bf6351a00c906fcc2b3d WatchSource:0}: Error finding container dff2fc763f3c28be5127113adbacdd95f34f94f19ca2bf6351a00c906fcc2b3d: Status 404 returned error can't find the container with id dff2fc763f3c28be5127113adbacdd95f34f94f19ca2bf6351a00c906fcc2b3d
Sep 11 04:23:10.629115 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.629093    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 04:23:10.649114 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.649094    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"]
Sep 11 04:23:10.650776 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:23:10.650758    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc98a94_4d24_42b8_aaf2_a28557b3d5c1.slice/crio-cb580b16190108626671a7ae32ceb24318059595def03fc8040d3bf00698b0bf WatchSource:0}: Error finding container cb580b16190108626671a7ae32ceb24318059595def03fc8040d3bf00698b0bf: Status 404 returned error can't find the container with id cb580b16190108626671a7ae32ceb24318059595def03fc8040d3bf00698b0bf
Sep 11 04:23:10.930080 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.929978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8" event={"ID":"52e19559-2350-40cb-8a17-b02cb4193830","Type":"ContainerStarted","Data":"dff2fc763f3c28be5127113adbacdd95f34f94f19ca2bf6351a00c906fcc2b3d"}
Sep 11 04:23:10.931238 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:10.931213    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms" event={"ID":"6cc98a94-4d24-42b8-aaf2-a28557b3d5c1","Type":"ContainerStarted","Data":"cb580b16190108626671a7ae32ceb24318059595def03fc8040d3bf00698b0bf"}
Sep 11 04:23:11.938349 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:11.938315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8" event={"ID":"52e19559-2350-40cb-8a17-b02cb4193830","Type":"ContainerStarted","Data":"d78210a06b3f7e7d2f695bd9d89b7e011795df20aa28398280c1edebd43048a9"}
Sep 11 04:23:11.938818 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:11.938380    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:11.939881 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:11.939849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms" event={"ID":"6cc98a94-4d24-42b8-aaf2-a28557b3d5c1","Type":"ContainerStarted","Data":"7913ce58a1901ddd8c7f1af2aac1bab487377128f909de76ddf4b0eedac647ed"}
Sep 11 04:23:11.940001 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:11.939890    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:11.955583 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:11.955524    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8" podStartSLOduration=1.477479855 podStartE2EDuration="1.955509186s" totalImagesPullingTime="478.029331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:23:11.954840674 +0000 UTC m=+1984.577233724" watchObservedRunningTime="2026-09-11 04:23:11.955509186 +0000 UTC m=+1984.577902213"
Sep 11 04:23:11.973781 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:11.973738    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms" podStartSLOduration=1.554326399 podStartE2EDuration="1.973724446s" totalImagesPullingTime="419.398047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:23:11.97270184 +0000 UTC m=+1984.595094895" watchObservedRunningTime="2026-09-11 04:23:11.973724446 +0000 UTC m=+1984.596117472"
Sep 11 04:23:17.949988 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:17.949953    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-9kgms"
Sep 11 04:23:17.950606 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:17.950334    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:18.018577 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.018545    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"]
Sep 11 04:23:18.018780 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.018755    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8" podUID="52e19559-2350-40cb-8a17-b02cb4193830" containerName="maas-api" containerID="cri-o://d78210a06b3f7e7d2f695bd9d89b7e011795df20aa28398280c1edebd43048a9" gracePeriod=30
Sep 11 04:23:18.265028 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.264995    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:18.437160 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.437133    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e19559-2350-40cb-8a17-b02cb4193830-maas-api-tls\") pod \"52e19559-2350-40cb-8a17-b02cb4193830\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") "
Sep 11 04:23:18.437355 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.437191    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e19559-2350-40cb-8a17-b02cb4193830-metrics-certs\") pod \"52e19559-2350-40cb-8a17-b02cb4193830\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") "
Sep 11 04:23:18.437355 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.437224    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dczs2\" (UniqueName: \"kubernetes.io/projected/52e19559-2350-40cb-8a17-b02cb4193830-kube-api-access-dczs2\") pod \"52e19559-2350-40cb-8a17-b02cb4193830\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") "
Sep 11 04:23:18.437355 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.437253    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e19559-2350-40cb-8a17-b02cb4193830-openshift-service-ca\") pod \"52e19559-2350-40cb-8a17-b02cb4193830\" (UID: \"52e19559-2350-40cb-8a17-b02cb4193830\") "
Sep 11 04:23:18.437665 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.437634    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e19559-2350-40cb-8a17-b02cb4193830-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52e19559-2350-40cb-8a17-b02cb4193830" (UID: "52e19559-2350-40cb-8a17-b02cb4193830"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:23:18.439298 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.439272    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e19559-2350-40cb-8a17-b02cb4193830-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52e19559-2350-40cb-8a17-b02cb4193830" (UID: "52e19559-2350-40cb-8a17-b02cb4193830"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:23:18.439421 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.439375    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e19559-2350-40cb-8a17-b02cb4193830-kube-api-access-dczs2" (OuterVolumeSpecName: "kube-api-access-dczs2") pod "52e19559-2350-40cb-8a17-b02cb4193830" (UID: "52e19559-2350-40cb-8a17-b02cb4193830"). InnerVolumeSpecName "kube-api-access-dczs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:23:18.439421 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.439410    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e19559-2350-40cb-8a17-b02cb4193830-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "52e19559-2350-40cb-8a17-b02cb4193830" (UID: "52e19559-2350-40cb-8a17-b02cb4193830"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:23:18.538439 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.538356    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e19559-2350-40cb-8a17-b02cb4193830-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:23:18.538439 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.538385    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e19559-2350-40cb-8a17-b02cb4193830-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:23:18.538439 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.538399    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dczs2\" (UniqueName: \"kubernetes.io/projected/52e19559-2350-40cb-8a17-b02cb4193830-kube-api-access-dczs2\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:23:18.538439 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.538408    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e19559-2350-40cb-8a17-b02cb4193830-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 11 04:23:18.976711 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.976679    2581 generic.go:356] "Generic (PLEG): container finished" podID="52e19559-2350-40cb-8a17-b02cb4193830" containerID="d78210a06b3f7e7d2f695bd9d89b7e011795df20aa28398280c1edebd43048a9" exitCode=0
Sep 11 04:23:18.977197 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.976726    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"
Sep 11 04:23:18.977262 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.976763    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8" event={"ID":"52e19559-2350-40cb-8a17-b02cb4193830","Type":"ContainerDied","Data":"d78210a06b3f7e7d2f695bd9d89b7e011795df20aa28398280c1edebd43048a9"}
Sep 11 04:23:18.977315 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.977263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8" event={"ID":"52e19559-2350-40cb-8a17-b02cb4193830","Type":"ContainerDied","Data":"dff2fc763f3c28be5127113adbacdd95f34f94f19ca2bf6351a00c906fcc2b3d"}
Sep 11 04:23:18.977371 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.977312    2581 scope.go:117] "RemoveContainer" containerID="d78210a06b3f7e7d2f695bd9d89b7e011795df20aa28398280c1edebd43048a9"
Sep 11 04:23:18.994223 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.994207    2581 scope.go:117] "RemoveContainer" containerID="d78210a06b3f7e7d2f695bd9d89b7e011795df20aa28398280c1edebd43048a9"
Sep 11 04:23:18.994482 ip-10-0-142-242 kubenswrapper[2581]: E0911 04:23:18.994460    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78210a06b3f7e7d2f695bd9d89b7e011795df20aa28398280c1edebd43048a9\": container with ID starting with d78210a06b3f7e7d2f695bd9d89b7e011795df20aa28398280c1edebd43048a9 not found: ID does not exist" containerID="d78210a06b3f7e7d2f695bd9d89b7e011795df20aa28398280c1edebd43048a9"
Sep 11 04:23:18.994554 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:18.994489    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78210a06b3f7e7d2f695bd9d89b7e011795df20aa28398280c1edebd43048a9"} err="failed to get container status \"d78210a06b3f7e7d2f695bd9d89b7e011795df20aa28398280c1edebd43048a9\": rpc error: code = NotFound desc = could not find container \"d78210a06b3f7e7d2f695bd9d89b7e011795df20aa28398280c1edebd43048a9\": container with ID starting with d78210a06b3f7e7d2f695bd9d89b7e011795df20aa28398280c1edebd43048a9 not found: ID does not exist"
Sep 11 04:23:19.002588 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:19.002565    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"]
Sep 11 04:23:19.008360 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:19.008337    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-cfc75856f-sqnk8"]
Sep 11 04:23:19.861949 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:19.861916    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52e19559-2350-40cb-8a17-b02cb4193830" path="/var/lib/kubelet/pods/52e19559-2350-40cb-8a17-b02cb4193830/volumes"
Sep 11 04:23:40.289426 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:40.289389    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-756b658b9-xz8t6_99b7a60f-140d-45c1-a886-ec51b9fc1375/authorino/0.log"
Sep 11 04:23:46.597503 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:46.597473    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97cf4878d-b54js_14dd9380-f197-4f2f-8918-5c0147c7eee5/copy-manifests/0.log"
Sep 11 04:23:46.604249 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:46.604224    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97cf4878d-b54js_14dd9380-f197-4f2f-8918-5c0147c7eee5/manager/0.log"
Sep 11 04:23:47.222251 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:47.222216    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6c9d99c6fc-9kgms_6cc98a94-4d24-42b8-aaf2-a28557b3d5c1/maas-api/0.log"
Sep 11 04:23:47.452777 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:47.452750    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818320-b9pmz_a96d5a3e-44ae-4d7b-920f-85f75446e414/cleanup/0.log"
Sep 11 04:23:48.528928 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:48.528897    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-756b658b9-xz8t6_99b7a60f-140d-45c1-a886-ec51b9fc1375/authorino/0.log"
Sep 11 04:23:48.763001 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:48.762972    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-8lppl_3003aee5-dd39-4517-9d87-d8fe2e824cc4/manager/0.log"
Sep 11 04:23:48.883067 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:48.883040    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-dx2tc_3a97df84-4421-44de-b006-a31aeefd96a7/kuadrant-console-plugin/0.log"
Sep 11 04:23:49.012585 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:49.012554    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-l727c_d52ae8a5-24b5-41f7-b7ca-060c409e6098/manager/0.log"
Sep 11 04:23:49.903241 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:49.903209    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5f7bb976-7h66z_16f0b644-f36e-474d-bb69-78158b5d9dfe/kube-auth-proxy/0.log"
Sep 11 04:23:51.027688 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:51.027658    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-tqdk7_eae38e05-6618-4768-9340-5e4a387551b9/main/0.log"
Sep 11 04:23:51.354671 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:51.354640    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jxpm_62213da7-5243-4289-aa68-9d57ca9f9e93/main/0.log"
Sep 11 04:23:57.696315 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:57.696277    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:23:57.775380 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:23:57.775343    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:24:00.622116 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:00.622082    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6bs2d_69e1dbc6-8f47-4733-9be1-d76b49f7f4d1/global-pull-secret-syncer/0.log"
Sep 11 04:24:00.764406 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:00.764367    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log"
Sep 11 04:24:00.895303 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:00.895227    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-242.ec2.internal_427abdc7e2784c3c5f6bbe0ee342a5b6/haproxy/0.log"
Sep 11 04:24:04.961067 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:04.961036    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-756b658b9-xz8t6_99b7a60f-140d-45c1-a886-ec51b9fc1375/authorino/0.log"
Sep 11 04:24:05.026816 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:05.026784    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-8lppl_3003aee5-dd39-4517-9d87-d8fe2e824cc4/manager/0.log"
Sep 11 04:24:05.073282 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:05.073258    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-dx2tc_3a97df84-4421-44de-b006-a31aeefd96a7/kuadrant-console-plugin/0.log"
Sep 11 04:24:05.133538 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:05.133511    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-l727c_d52ae8a5-24b5-41f7-b7ca-060c409e6098/manager/0.log"
Sep 11 04:24:06.726474 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:06.726448    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7cb85423-2c36-4080-93d4-432b6df777e4/alertmanager/0.log"
Sep 11 04:24:06.768329 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:06.768303    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7cb85423-2c36-4080-93d4-432b6df777e4/config-reloader/0.log"
Sep 11 04:24:06.798545 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:06.798517    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7cb85423-2c36-4080-93d4-432b6df777e4/kube-rbac-proxy-web/0.log"
Sep 11 04:24:06.825860 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:06.825839    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7cb85423-2c36-4080-93d4-432b6df777e4/kube-rbac-proxy/0.log"
Sep 11 04:24:06.853208 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:06.853175    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7cb85423-2c36-4080-93d4-432b6df777e4/kube-rbac-proxy-metric/0.log"
Sep 11 04:24:06.880941 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:06.880917    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7cb85423-2c36-4080-93d4-432b6df777e4/prom-label-proxy/0.log"
Sep 11 04:24:06.911877 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:06.911859    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7cb85423-2c36-4080-93d4-432b6df777e4/init-config-reloader/0.log"
Sep 11 04:24:07.036983 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.036904    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-c2dtj_3e3faf4d-01be-49e9-99e3-227ad2618ffc/kube-state-metrics/0.log"
Sep 11 04:24:07.075455 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.075428    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-c2dtj_3e3faf4d-01be-49e9-99e3-227ad2618ffc/kube-rbac-proxy-main/0.log"
Sep 11 04:24:07.104910 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.104886    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-c2dtj_3e3faf4d-01be-49e9-99e3-227ad2618ffc/kube-rbac-proxy-self/0.log"
Sep 11 04:24:07.139380 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.139358    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-bb4598894-mcwhb_d5fcb49d-518f-4960-9ab6-33155afc702b/metrics-server/0.log"
Sep 11 04:24:07.173251 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.173232    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-6qnsv_fa49c5a7-80aa-4a35-b4fe-30a674060be7/monitoring-plugin/0.log"
Sep 11 04:24:07.212485 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.212463    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbpcl_57c8a052-e86a-4c05-bf59-d622c4962c84/node-exporter/0.log"
Sep 11 04:24:07.249608 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.249585    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbpcl_57c8a052-e86a-4c05-bf59-d622c4962c84/kube-rbac-proxy/0.log"
Sep 11 04:24:07.281765 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.281741    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbpcl_57c8a052-e86a-4c05-bf59-d622c4962c84/init-textfile/0.log"
Sep 11 04:24:07.386988 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.386962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qtbp_0ca873b5-3619-4087-a38d-221452120a26/konnectivity-agent/0.log" limit=1024
Sep 11 04:24:07.686547 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.686520    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4qt8h_7e663698-55a2-49d2-9c40-136d0cf7c9be/kube-rbac-proxy-main/0.log"
Sep 11 04:24:07.750919 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.750891    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4qt8h_7e663698-55a2-49d2-9c40-136d0cf7c9be/kube-rbac-proxy-self/0.log"
Sep 11 04:24:07.814484 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.814460    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4qt8h_7e663698-55a2-49d2-9c40-136d0cf7c9be/openshift-state-metrics/0.log"
Sep 11 04:24:07.899447 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.899415    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78d1ed64-d42b-4b80-a0ec-68f904e00b11/prometheus/0.log"
Sep 11 04:24:07.939282 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.939197    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78d1ed64-d42b-4b80-a0ec-68f904e00b11/config-reloader/0.log"
Sep 11 04:24:07.976184 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:07.976163    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78d1ed64-d42b-4b80-a0ec-68f904e00b11/thanos-sidecar/0.log"
Sep 11 04:24:08.011715 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.011690    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78d1ed64-d42b-4b80-a0ec-68f904e00b11/kube-rbac-proxy-web/0.log"
Sep 11 04:24:08.041827 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.041801    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78d1ed64-d42b-4b80-a0ec-68f904e00b11/kube-rbac-proxy/0.log"
Sep 11 04:24:08.077027 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.077006    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78d1ed64-d42b-4b80-a0ec-68f904e00b11/kube-rbac-proxy-thanos/0.log"
Sep 11 04:24:08.121723 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.121702    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78d1ed64-d42b-4b80-a0ec-68f904e00b11/init-config-reloader/0.log"
Sep 11 04:24:08.168226 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.168203    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-hmfjg_fe31f333-a4cb-49b4-b509-05a46d249115/prometheus-operator/0.log"
Sep 11 04:24:08.198558 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.198502    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-hmfjg_fe31f333-a4cb-49b4-b509-05a46d249115/kube-rbac-proxy/0.log"
Sep 11 04:24:08.233289 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.233265    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-mn7sm_f1ad1d47-9222-4867-9fa5-91966c1317d3/prometheus-operator-admission-webhook/0.log"
Sep 11 04:24:08.278090 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.278068    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fc8677df8-cplcz_01dec1b6-1add-4f76-8e6b-eb91561ebbc8/telemeter-client/0.log"
Sep 11 04:24:08.312774 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.312754    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fc8677df8-cplcz_01dec1b6-1add-4f76-8e6b-eb91561ebbc8/reload/0.log"
Sep 11 04:24:08.342843 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.342820    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fc8677df8-cplcz_01dec1b6-1add-4f76-8e6b-eb91561ebbc8/kube-rbac-proxy/0.log"
Sep 11 04:24:08.384318 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.384296    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b4b66dbdb-q8bdv_806bef0d-627b-423d-8929-1ae0f2701a5a/thanos-query/0.log"
Sep 11 04:24:08.412474 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.412450    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b4b66dbdb-q8bdv_806bef0d-627b-423d-8929-1ae0f2701a5a/kube-rbac-proxy-web/0.log"
Sep 11 04:24:08.441702 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.441681    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b4b66dbdb-q8bdv_806bef0d-627b-423d-8929-1ae0f2701a5a/kube-rbac-proxy/0.log"
Sep 11 04:24:08.486175 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.486101    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b4b66dbdb-q8bdv_806bef0d-627b-423d-8929-1ae0f2701a5a/prom-label-proxy/0.log"
Sep 11 04:24:08.548590 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.548557    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b4b66dbdb-q8bdv_806bef0d-627b-423d-8929-1ae0f2701a5a/kube-rbac-proxy-rules/0.log"
Sep 11 04:24:08.594132 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:08.594106    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b4b66dbdb-q8bdv_806bef0d-627b-423d-8929-1ae0f2701a5a/kube-rbac-proxy-metrics/0.log"
Sep 11 04:24:10.212087 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.212050    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"]
Sep 11 04:24:10.212664 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.212644    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52e19559-2350-40cb-8a17-b02cb4193830" containerName="maas-api"
Sep 11 04:24:10.212743 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.212668    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="52e19559-2350-40cb-8a17-b02cb4193830" containerName="maas-api"
Sep 11 04:24:10.212830 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.212817    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="52e19559-2350-40cb-8a17-b02cb4193830" containerName="maas-api"
Sep 11 04:24:10.217505 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.217483    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.220012 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.219990    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vb5nd\"/\"kube-root-ca.crt\""
Sep 11 04:24:10.220416 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.220060    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vb5nd\"/\"openshift-service-ca.crt\""
Sep 11 04:24:10.221163 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.221143    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vb5nd\"/\"default-dockercfg-pfqxn\""
Sep 11 04:24:10.222697 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.222676    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"]
Sep 11 04:24:10.322056 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.322003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5pxv\" (UniqueName: \"kubernetes.io/projected/a5e8f894-3636-42d4-afad-20ae770b8209-kube-api-access-g5pxv\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.322056 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.322060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5e8f894-3636-42d4-afad-20ae770b8209-proc\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.322260 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.322095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5e8f894-3636-42d4-afad-20ae770b8209-podres\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.322260 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.322141    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5e8f894-3636-42d4-afad-20ae770b8209-lib-modules\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.322260 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.322180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5e8f894-3636-42d4-afad-20ae770b8209-sys\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.423423 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.423391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5pxv\" (UniqueName: \"kubernetes.io/projected/a5e8f894-3636-42d4-afad-20ae770b8209-kube-api-access-g5pxv\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.423423 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.423431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5e8f894-3636-42d4-afad-20ae770b8209-proc\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.423690 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.423447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5e8f894-3636-42d4-afad-20ae770b8209-podres\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.423690 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.423490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5e8f894-3636-42d4-afad-20ae770b8209-lib-modules\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.423690 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.423540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5e8f894-3636-42d4-afad-20ae770b8209-proc\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.423690 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.423549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5e8f894-3636-42d4-afad-20ae770b8209-sys\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.423690 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.423597    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5e8f894-3636-42d4-afad-20ae770b8209-sys\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.423690 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.423658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5e8f894-3636-42d4-afad-20ae770b8209-podres\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.423690 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.423672    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5e8f894-3636-42d4-afad-20ae770b8209-lib-modules\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.432200 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.432172    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5pxv\" (UniqueName: \"kubernetes.io/projected/a5e8f894-3636-42d4-afad-20ae770b8209-kube-api-access-g5pxv\") pod \"perf-node-gather-daemonset-vzgtt\" (UID: \"a5e8f894-3636-42d4-afad-20ae770b8209\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.529245 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.529177    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:10.799003 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.798980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"]
Sep 11 04:24:10.801225 ip-10-0-142-242 kubenswrapper[2581]: W0911 04:24:10.801197    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda5e8f894_3636_42d4_afad_20ae770b8209.slice/crio-d76a65517ae73c77d533452dd0eec5eead2ecf3036141fe65df3ba0784949577 WatchSource:0}: Error finding container d76a65517ae73c77d533452dd0eec5eead2ecf3036141fe65df3ba0784949577: Status 404 returned error can't find the container with id d76a65517ae73c77d533452dd0eec5eead2ecf3036141fe65df3ba0784949577
Sep 11 04:24:10.923393 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:10.923364    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c569b9647-d4pk9_fdff324d-d735-4f06-9dd9-2a31b4a2e597/console/0.log"
Sep 11 04:24:11.245413 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:11.245377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt" event={"ID":"a5e8f894-3636-42d4-afad-20ae770b8209","Type":"ContainerStarted","Data":"1017d7c1b2b2baeaf77127f4053b3aa8c04666f144351d49432ccf07a0773939"}
Sep 11 04:24:11.245778 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:11.245453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt" event={"ID":"a5e8f894-3636-42d4-afad-20ae770b8209","Type":"ContainerStarted","Data":"d76a65517ae73c77d533452dd0eec5eead2ecf3036141fe65df3ba0784949577"}
Sep 11 04:24:11.245778 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:11.245590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:11.263256 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:11.263209    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt" podStartSLOduration=1.2631962350000001 podStartE2EDuration="1.263196235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 04:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:24:11.260962919 +0000 UTC m=+2043.883355968" watchObservedRunningTime="2026-09-11 04:24:11.263196235 +0000 UTC m=+2043.885589262"
Sep 11 04:24:12.673426 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:12.673388    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5gtpq_1e7b003d-b66d-4eef-8738-29ae74b53cc5/dns/0.log"
Sep 11 04:24:12.694658 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:12.694628    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5gtpq_1e7b003d-b66d-4eef-8738-29ae74b53cc5/kube-rbac-proxy/0.log"
Sep 11 04:24:12.787684 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:12.787657    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4cm6g_5604fe2c-0315-490c-a234-cae850ae2d55/dns-node-resolver/0.log"
Sep 11 04:24:13.403960 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:13.403928    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mkxx2_657845f4-5ec8-4c63-9d6a-f9ca4273be0e/node-ca/0.log"
Sep 11 04:24:14.578284 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:14.578254    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5f7bb976-7h66z_16f0b644-f36e-474d-bb69-78158b5d9dfe/kube-auth-proxy/0.log"
Sep 11 04:24:15.551064 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:15.551016    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-65vbf_40496107-347a-4a7d-bf59-25f0f8ac4a41/serve-healthcheck-canary/0.log"
Sep 11 04:24:16.127823 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:16.127793    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-krd62_df2bd540-5d8f-4eaf-93da-97ad522413e0/kube-rbac-proxy/0.log"
Sep 11 04:24:16.147437 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:16.147416    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-krd62_df2bd540-5d8f-4eaf-93da-97ad522413e0/exporter/0.log"
Sep 11 04:24:16.168656 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:16.168630    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-krd62_df2bd540-5d8f-4eaf-93da-97ad522413e0/extractor/0.log"
Sep 11 04:24:17.260269 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:17.260245    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-vzgtt"
Sep 11 04:24:18.646601 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:18.646554    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97cf4878d-b54js_14dd9380-f197-4f2f-8918-5c0147c7eee5/manager/0.log"
Sep 11 04:24:18.673042 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:18.673002    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97cf4878d-b54js_14dd9380-f197-4f2f-8918-5c0147c7eee5/copy-manifests/0.log"
Sep 11 04:24:20.000633 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:20.000601    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-67f66d78bd-4fmbs_4fbdbefa-89ef-46b7-ba15-e566c16a5071/manager/0.log"
Sep 11 04:24:20.175459 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:20.175384    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5f7bb976-7h66z_16f0b644-f36e-474d-bb69-78158b5d9dfe/kube-auth-proxy/0.log"
Sep 11 04:24:22.076474 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:22.076445    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5f7bb976-7h66z_16f0b644-f36e-474d-bb69-78158b5d9dfe/kube-auth-proxy/0.log"
Sep 11 04:24:26.664467 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:26.664437    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tvdds_f5ad581b-7f8b-4747-8aad-7b59824edac5/kube-multus-additional-cni-plugins/0.log"
Sep 11 04:24:26.684650 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:26.684615    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tvdds_f5ad581b-7f8b-4747-8aad-7b59824edac5/egress-router-binary-copy/0.log"
Sep 11 04:24:26.705168 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:26.705140    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tvdds_f5ad581b-7f8b-4747-8aad-7b59824edac5/cni-plugins/0.log"
Sep 11 04:24:26.725614 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:26.725590    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tvdds_f5ad581b-7f8b-4747-8aad-7b59824edac5/bond-cni-plugin/0.log"
Sep 11 04:24:26.744612 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:26.744592    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tvdds_f5ad581b-7f8b-4747-8aad-7b59824edac5/routeoverride-cni/0.log"
Sep 11 04:24:26.763959 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:26.763933    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tvdds_f5ad581b-7f8b-4747-8aad-7b59824edac5/whereabouts-cni-bincopy/0.log"
Sep 11 04:24:26.787125 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:26.787104    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tvdds_f5ad581b-7f8b-4747-8aad-7b59824edac5/whereabouts-cni/0.log"
Sep 11 04:24:27.043035 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:27.042944    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vqrfz_443eee6f-26ee-439c-bd85-2bc0b92fdd14/kube-multus/0.log"
Sep 11 04:24:27.110483 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:27.110453    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6rpfl_af6ebd1b-3025-4a6f-9822-aa6606b8fa28/network-metrics-daemon/0.log"
Sep 11 04:24:27.128653 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:27.128627    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6rpfl_af6ebd1b-3025-4a6f-9822-aa6606b8fa28/kube-rbac-proxy/0.log"
Sep 11 04:24:28.032399 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:28.032363    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxqls_8f89a78c-3f9c-45bf-a1dc-3544ed06740f/ovn-controller/0.log"
Sep 11 04:24:28.048554 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:28.048526    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxqls_8f89a78c-3f9c-45bf-a1dc-3544ed06740f/ovn-acl-logging/0.log"
Sep 11 04:24:28.072117 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:28.072090    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxqls_8f89a78c-3f9c-45bf-a1dc-3544ed06740f/kube-rbac-proxy-node/0.log"
Sep 11 04:24:28.097179 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:28.097160    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxqls_8f89a78c-3f9c-45bf-a1dc-3544ed06740f/kube-rbac-proxy-ovn-metrics/0.log"
Sep 11 04:24:28.118625 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:28.118599    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxqls_8f89a78c-3f9c-45bf-a1dc-3544ed06740f/northd/0.log"
Sep 11 04:24:28.140531 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:28.140515    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxqls_8f89a78c-3f9c-45bf-a1dc-3544ed06740f/nbdb/0.log"
Sep 11 04:24:28.166938 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:28.166918    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxqls_8f89a78c-3f9c-45bf-a1dc-3544ed06740f/sbdb/0.log"
Sep 11 04:24:28.252768 ip-10-0-142-242 kubenswrapper[2581]: I0911 04:24:28.252735    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxqls_8f89a78c-3f9c-45bf-a1dc-3544ed06740f/ovnkube-controller/0.log"