Sep 10 19:54:17.690383 ip-10-0-140-111 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 19:54:18.082649 ip-10-0-140-111 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 19:54:18.082649 ip-10-0-140-111 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 19:54:18.082649 ip-10-0-140-111 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 19:54:18.082649 ip-10-0-140-111 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 19:54:18.082649 ip-10-0-140-111 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 19:54:18.083398 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.083307    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 19:54:18.086501 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086481    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:54:18.086501 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086500    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:54:18.086501 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086504    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086507    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086511    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086513    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086518    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086521    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086524    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086527    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086530    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086532    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086535    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086537    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086540    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086543    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086545    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086548    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086550    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086553    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086570    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086573    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:54:18.086619 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086575    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086578    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086581    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086583    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086586    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086588    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086591    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086593    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086596    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086598    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086602    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086604    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086606    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086609    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086613    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086616    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086618    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086621    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086624    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086626    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:54:18.087103 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086629    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086634    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086637    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086640    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086643    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086646    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086649    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086652    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086654    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086657    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086660    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086664    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086666    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086669    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086671    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086676    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086680    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086683    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086686    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:54:18.087622 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086689    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086692    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086694    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086697    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086700    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086704    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086707    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086710    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086713    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086716    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086718    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086721    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086724    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086726    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086729    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086732    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086735    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086738    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086740    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086742    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086745    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:54:18.088096 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086747    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086750    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086752    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086755    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086758    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086760    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086763    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086765    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086768    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086770    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086773    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086775    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086778    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086782    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086786    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086789    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.086792    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087595    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087604    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087608    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:54:18.088590 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087611    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087613    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087616    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087619    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087622    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087625    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087629    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087631    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087634    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087637    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087639    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087642    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087645    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087648    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087651    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087653    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087656    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087659    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087662    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087664    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:54:18.089080 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087667    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087669    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087672    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087675    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087677    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087680    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087682    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087685    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087687    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087689    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087693    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087695    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087698    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087700    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087703    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087706    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087709    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087712    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087714    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:54:18.089573 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087717    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087719    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087721    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087724    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087727    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087729    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087731    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087734    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087736    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087739    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087741    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087744    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087747    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087749    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087751    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087754    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087758    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087761    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087763    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087766    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:54:18.090104 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087768    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087772    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087774    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087778    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087780    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087783    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087785    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087788    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087790    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087793    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087797    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087801    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087804    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087807    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087810    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087813    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087815    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087818    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087820    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:54:18.090598 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087823    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087825    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087828    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087831    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087834    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087837    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087841    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087845    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087848    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087851    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087853    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087855    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087858    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087861    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087864    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087866    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087869    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.087872    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088322    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088328    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:54:18.091070 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088331    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088334    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088336    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088339    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088342    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088345    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088347    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088350    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088352    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088355    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088357    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088360    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088363    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088366    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088368    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088371    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088373    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088377    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088379    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088382    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:54:18.091564 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088385    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088388    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088390    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088393    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088396    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088399    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088403    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088406    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088409    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088411    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088414    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088416    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088419    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088421    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088424    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088426    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088429    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088431    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088434    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:54:18.092053 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088436    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088439    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088441    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088444    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088446    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088449    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088451    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088454    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088456    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088460    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088462    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088465    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088468    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088470    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088473    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088476    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088478    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088481    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088484    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088487    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:54:18.092512 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088489    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088492    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088495    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088497    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088500    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088502    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088507    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088510    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088514    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088517    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088520    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088522    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088525    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088527    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088530    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088532    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088534    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088537    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088540    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088542    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:54:18.092985 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088544    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088547    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088549    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088552    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088555    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088557    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088560    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088562    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088565    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088567    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088569    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088572    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088574    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088583    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088585    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088589    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088593    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.088595    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088679    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088735    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 19:54:18.093502 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088746    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088751    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088755    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088758    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088762    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088767    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088770    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088773    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088776    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088780    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088783    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088786    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088788    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088791    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088794    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088797    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088802    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088805    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088808    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088812    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088815    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088819    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088822    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 19:54:18.093983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088825    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088829    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088833    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088836    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088839    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088842    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088845    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088849    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088852    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088855    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088858    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088862    2561 flags.go:64] FLAG: --enable-server="true"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088865    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088869    2561 flags.go:64] FLAG: --event-burst="100"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088872    2561 flags.go:64] FLAG: --event-qps="50"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088875    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088878    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088881    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088885    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088888    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088891    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088894    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088897    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088901    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088904    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 19:54:18.094570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088907    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088910    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088913    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088916    2561 flags.go:64] FLAG: --feature-gates=""
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088921    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088924    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088927    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088930    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088933    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088937    2561 flags.go:64] FLAG: --help="false"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088940    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088943    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088946    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088949    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088953    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088956    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088960    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088963    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088965    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088968    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088971    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088974    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.088977    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089021    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 19:54:18.095231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089025    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089045    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089049    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089052    2561 flags.go:64] FLAG: --lock-file=""
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089055    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089058    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089061    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089069    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089072    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089075    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089078    2561 flags.go:64] FLAG: --logging-format="text"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089082    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089085    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089088    2561 flags.go:64] FLAG: --manifest-url=""
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089091    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089096    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089099    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089104    2561 flags.go:64] FLAG: --max-pods="110"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089107    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089110    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089113    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089122    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089126    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089129    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089133    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 19:54:18.095854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089141    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089144    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089147    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089150    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089153    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089158    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089161    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089165    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089168    2561 flags.go:64] FLAG: --port="10250"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089171    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089174    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-094990131f5a7f752"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089177    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089180    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089183    2561 flags.go:64] FLAG: --register-node="true"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089186    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089192    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089195    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089198    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089200    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089210    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089214    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089217    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089220    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089223    2561 flags.go:64] FLAG: --runonce="false"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089226    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 19:54:18.096903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089229    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089232    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089235    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089238    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089241    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089244    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089247    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089250    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089253    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089255    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089258    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089261    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089264    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089267    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089272    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089275    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089279    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089286    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089289    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089292    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089294    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089297    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089301    2561 flags.go:64] FLAG: --v="2"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089306    2561 flags.go:64] FLAG: --version="false"
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089310    2561 flags.go:64] FLAG: --vmodule=""
Sep 10 19:54:18.097962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089314    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089317    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089490    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089494    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089498    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089500    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089503    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089505    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089508    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089511    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089514    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089516    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089519    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089522    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089524    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089527    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089529    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089533    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089537    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:54:18.099288 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089540    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089543    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089546    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089549    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089553    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089556    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089558    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089560    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089563    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089565    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089569    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089572    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089574    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089577    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089580    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089584    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089587    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089589    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089592    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089595    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:54:18.100128 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089597    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089600    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089602    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089604    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089607    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089609    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089612    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089614    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089616    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089619    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089622    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089624    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089627    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089629    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089632    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089634    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089638    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089640    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089643    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:54:18.100650 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089645    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089648    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089650    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089654    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089657    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089659    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089662    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089664    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089667    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089671    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089677    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089680    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089683    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089685    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089688    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089691    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089694    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089698    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089701    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089704    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:54:18.101488 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089706    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089709    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089711    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089714    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089717    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089720    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089722    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089725    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089727    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089731    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089733    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089736    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089738    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089741    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089743    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089747    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089750    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089752    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089755    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089758    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:54:18.102205 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089760    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089763    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.089765    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.089774    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.098429    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.098458    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.098482    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.098491    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098589    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098597    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098602    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098606    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098610    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098614    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098618    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:54:18.102730 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098622    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098626    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098631    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098635    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098639    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098643    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098648    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098651    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098655    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098660    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098664    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098668    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098672    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098676    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098680    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098685    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098689    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098693    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098697    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098701    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:54:18.103126 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098705    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098709    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098721    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098726    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098730    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098734    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098738    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098742    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098746    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098750    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098754    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098758    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098762    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098766    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098770    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098774    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098778    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098782    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098786    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098790    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:54:18.103637 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098794    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098798    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098802    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098806    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098809    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098813    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098817    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098821    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098825    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098829    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098835    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098841    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098846    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098851    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098857    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098869    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098873    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098877    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098881    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:54:18.104148 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098885    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098889    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098893    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098898    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098902    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098906    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098910    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098914    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098917    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098921    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098925    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098929    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098933    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098939    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098945    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098951    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098955    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098959    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098963    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:54:18.104647 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098967    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098971    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098975    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098979    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098983    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098987    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098991    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.098995    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099000    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099004    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099016    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099021    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099027    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099051    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.099059    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 19:54:18.105155 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099266    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099274    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099279    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099283    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099287    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099292    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099295    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099299    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099303    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099307    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099310    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099314    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099318    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099322    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099327    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099333    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099338    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099342    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099346    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:54:18.105614 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099350    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099354    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099358    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099362    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099367    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099371    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099375    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099380    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099384    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099397    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099401    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099405    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099409    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099413    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099417    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099423    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099429    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099434    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099439    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:54:18.106277 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099444    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099448    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099453    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099457    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099461    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099465    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099470    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099474    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099478    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099482    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099485    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099489    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099494    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099498    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099501    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099505    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099509    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099514    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099521    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:54:18.106777 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099526    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099531    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099535    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099540    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099551    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099556    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099560    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099564    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099568    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099572    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099576    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099580    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099584    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099588    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099592    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099596    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099600    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099604    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099608    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099612    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:54:18.107363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099615    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099619    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099623    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099627    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099631    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099635    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099639    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099643    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099646    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099650    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099654    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099658    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099662    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099666    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099671    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099675    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099678    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099689    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099694    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099698    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099703    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:54:18.107915 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:18.099707    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:54:18.108476 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.099714    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 19:54:18.108476 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.099937    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 19:54:18.108476 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.106671    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 19:54:18.108476 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.107678    2561 server.go:1019] "Starting client certificate rotation"
Sep 10 19:54:18.108476 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.107791    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 19:54:18.108476 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.107839    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 19:54:18.110489 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.110469    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 19:54:18.114276 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.114252    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 19:54:18.128344 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.128323    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 10 19:54:18.131500 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.131483    2561 log.go:25] "Validated CRI v1 image API"
Sep 10 19:54:18.132435 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.132416    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 19:54:18.138237 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.138214    2561 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 f12b9be0-1b80-4c34-bb69-0fbc28ca14cb:/dev/nvme0n1p3 f45b7b4c-289c-460e-9254-1946babfc9bf:/dev/nvme0n1p4]
Sep 10 19:54:18.138314 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.138235    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 19:54:18.140899 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.140876    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 19:54:18.144025 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.143911    2561 manager.go:216] Machine: {Timestamp:2026-09-10 19:54:18.143357786 +0000 UTC m=+0.339379633 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090251 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2beff49c73a75cc3dc9dfb2ea675e0 SystemUUID:ec2beff4-9c73-a75c-c3dc-9dfb2ea675e0 BootID:cd074fd7-1e06-4042-bbd7-2ed59b09ce6b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4b:bf:28:15:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4b:bf:28:15:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:a4:f6:ff:68:27 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 19:54:18.144025 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.144018    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 19:54:18.144150 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.144118    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 19:54:18.147658 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.147630    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 19:54:18.147815 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.147660    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-111.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 19:54:18.149391 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.149380    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 19:54:18.149432 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.149394    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 19:54:18.149432 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.149402    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 19:54:18.150327 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.150315    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 19:54:18.152156 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.152144    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 19:54:18.156543 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.156523    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 19:54:18.156684 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.156674    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 19:54:18.156721 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.156710    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 19:54:18.156752 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.156723    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 19:54:18.156752 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.156739    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 19:54:18.156752 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.156749    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 10 19:54:18.156854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.156758    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 19:54:18.157144 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.157133    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 19:54:18.157197 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.157152    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 19:54:18.161384 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.161368    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 19:54:18.164062 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164048    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 19:54:18.164106 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164079    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 19:54:18.164253 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164244    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 19:54:18.164285 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164259    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 19:54:18.164285 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164265    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 19:54:18.164285 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164272    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 19:54:18.164285 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164281    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 19:54:18.164399 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164290    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 19:54:18.164841 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164830    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 19:54:18.164841 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164839    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 19:54:18.164902 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164848    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 19:54:18.164902 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164853    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 19:54:18.164902 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164870    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 19:54:18.164902 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164878    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 19:54:18.165009 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164911    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 19:54:18.165009 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.164917    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 19:54:18.168683 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.168669    2561 server.go:1268] "Started kubelet"
Sep 10 19:54:18.168798 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.168750    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 19:54:18.168909 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.168864    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 19:54:18.168967 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.168930    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 19:54:18.169212 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.169195    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-111.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 19:54:18.169296 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.169275    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 19:54:18.169969 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.169948    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 19:54:18.170402 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.170379    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 19:54:18.170762 ip-10-0-140-111 systemd[1]: Started Kubernetes Kubelet.
Sep 10 19:54:18.173483 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.173451    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xk94q"
Sep 10 19:54:18.174399 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.174372    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 19:54:18.177509 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.177488    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 19:54:18.178647 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.178622    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 19:54:18.178891 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.178851    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 19:54:18.178957 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.178937    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:18.179085 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.175288    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-111.ec2.internal.18d40dbd2f6e0267  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-111.ec2.internal,UID:ip-10-0-140-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-111.ec2.internal,},FirstTimestamp:2026-09-10 19:54:18.168648295 +0000 UTC m=+0.364670142,LastTimestamp:2026-09-10 19:54:18.168648295 +0000 UTC m=+0.364670142,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-111.ec2.internal,}"
Sep 10 19:54:18.179879 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.179854    2561 factory.go:55] Registering systemd factory
Sep 10 19:54:18.179879 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.179878    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 10 19:54:18.180019 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.179926    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 19:54:18.180019 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.179985    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 19:54:18.180019 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.179999    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 19:54:18.180140 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.180094    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 19:54:18.180140 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.180104    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 19:54:18.180140 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.180125    2561 factory.go:153] Registering CRI-O factory
Sep 10 19:54:18.180140 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.180136    2561 factory.go:223] Registration of the crio container factory successfully
Sep 10 19:54:18.180294 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.180188    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 19:54:18.180294 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.180207    2561 factory.go:103] Registering Raw factory
Sep 10 19:54:18.180294 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.180232    2561 manager.go:1196] Started watching for new ooms in manager
Sep 10 19:54:18.180613 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.180600    2561 manager.go:349] Starting recovery of all containers
Sep 10 19:54:18.181534 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.181508    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xk94q"
Sep 10 19:54:18.186968 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.186946    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 19:54:18.191193 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.191171    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:54:18.194395 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.194207    2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-111.ec2.internal\" not found" node="ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.197536 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.197517    2561 manager.go:354] Recovery completed
Sep 10 19:54:18.202392 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.202379    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:54:18.205134 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.205117    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:54:18.205202 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.205150    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:54:18.205202 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.205162    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:54:18.205781 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.205763    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 19:54:18.205781 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.205778    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 19:54:18.205883 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.205766    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:54:18.205883 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.205799    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 19:54:18.206627 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.206606    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:54:18.206719 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.206645    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:54:18.206719 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.206663    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:54:18.207888 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.207874    2561 policy_none.go:49] "None policy: Start"
Sep 10 19:54:18.207948 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.207894    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 19:54:18.207948 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.207907    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 19:54:18.209854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.209841    2561 policy_none.go:47] "Start"
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.246747    2561 manager.go:337] "Starting Device Plugin manager"
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.246785    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.246794    2561 server.go:87] "Starting device plugin registration server"
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.247178    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.247200    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.247284    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.247364    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.247371    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.247890    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.247934    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.251805    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.251850    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.251869    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.251944    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 19:54:18.262656 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.254757    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:54:18.348062 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.347933    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:54:18.348976 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.348960    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:54:18.349026 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.348993    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:54:18.349026 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.349008    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:54:18.349106 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.349054    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.352241 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.352206    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal"]
Sep 10 19:54:18.352300 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.352293    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:54:18.353092 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.353076    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:54:18.353169 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.353103    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:54:18.353169 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.353112    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:54:18.355246 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.355234    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:54:18.355391 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.355376    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.355435 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.355408    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:54:18.355876 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.355859    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.355876 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.355875    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-111.ec2.internal\": node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:18.356021 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.355947    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:54:18.356021 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.355974    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:54:18.356021 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.355985    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:54:18.356021 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.355953    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:54:18.356021 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.356010    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:54:18.356214 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.356052    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:54:18.359177 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.359161    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.359239 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.359191    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:54:18.359910 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.359887    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:54:18.359989 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.359917    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:54:18.359989 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.359928    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:54:18.373273 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.373250    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:18.374714 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.374699    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-111.ec2.internal\" not found" node="ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.378790 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.378773    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-111.ec2.internal\" not found" node="ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.473700 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.473672    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:18.480975 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.480951    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/534d538bb166d56e68f0f739a61e6155-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"534d538bb166d56e68f0f739a61e6155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.481060 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.480981    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/534d538bb166d56e68f0f739a61e6155-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"534d538bb166d56e68f0f739a61e6155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.481060 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.481000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10643e86bc3da5fa40e6977a1909b44c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-111.ec2.internal\" (UID: \"10643e86bc3da5fa40e6977a1909b44c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.574369 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.574341    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:18.581709 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.581685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10643e86bc3da5fa40e6977a1909b44c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-111.ec2.internal\" (UID: \"10643e86bc3da5fa40e6977a1909b44c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.581843 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.581722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/534d538bb166d56e68f0f739a61e6155-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"534d538bb166d56e68f0f739a61e6155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.581843 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.581745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/534d538bb166d56e68f0f739a61e6155-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"534d538bb166d56e68f0f739a61e6155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.581843 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.581803    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/534d538bb166d56e68f0f739a61e6155-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"534d538bb166d56e68f0f739a61e6155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.581843 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.581804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10643e86bc3da5fa40e6977a1909b44c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-111.ec2.internal\" (UID: \"10643e86bc3da5fa40e6977a1909b44c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.581983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.581837    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/534d538bb166d56e68f0f739a61e6155-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"534d538bb166d56e68f0f739a61e6155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.675239 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.675161    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:18.678328 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.678307    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.682292 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:18.682274    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:18.776333 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.776290    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:18.876858 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.876834    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:18.977423 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:18.977346    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:19.078011 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:19.077974    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:19.107473 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:19.107445    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 19:54:19.107903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:19.107596    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 19:54:19.107903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:19.107622    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 19:54:19.178604 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:19.178572    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:19.184178 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:19.184144    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 19:49:19 +0000 UTC" deadline="2028-03-19 00:07:39.407874492 +0000 UTC"
Sep 10 19:54:19.184328 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:19.184171    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13324h13m20.223706912s"
Sep 10 19:54:19.278911 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:19.278837    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:19.351178 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:19.351140    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10643e86bc3da5fa40e6977a1909b44c.slice/crio-3eec1dbd306fcaa0868cb4f3ccb32a0c4d383da913a75724663735dca798350b WatchSource:0}: Error finding container 3eec1dbd306fcaa0868cb4f3ccb32a0c4d383da913a75724663735dca798350b: Status 404 returned error can't find the container with id 3eec1dbd306fcaa0868cb4f3ccb32a0c4d383da913a75724663735dca798350b
Sep 10 19:54:19.351414 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:19.351394    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534d538bb166d56e68f0f739a61e6155.slice/crio-b91973ed19da46e5a7c3c5156832af2a9ae00531b3dff8881402dab052882954 WatchSource:0}: Error finding container b91973ed19da46e5a7c3c5156832af2a9ae00531b3dff8881402dab052882954: Status 404 returned error can't find the container with id b91973ed19da46e5a7c3c5156832af2a9ae00531b3dff8881402dab052882954
Sep 10 19:54:19.356403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:19.356270    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 19:54:19.378964 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:19.378926    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:19.479421 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:19.479380    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found"
Sep 10 19:54:19.496004 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:19.495980    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:54:19.536212 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:19.536127    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:54:19.580128 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:19.580099    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:19.592237 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:19.592214    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 19:54:19.593400 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:19.593385    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal"
Sep 10 19:54:19.602101 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:19.602079    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 19:54:21.304267 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.304236    2561 apiserver.go:52] "Watching apiserver"
Sep 10 19:54:21.313805 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.313738    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 19:54:21.314199 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.314170    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-99z2f","openshift-dns/node-resolver-tb2sv","openshift-multus/multus-additional-cni-plugins-6n4sz","openshift-network-operator/iptables-alerter-cbcc9","openshift-ovn-kubernetes/ovnkube-node-xstzr","kube-system/global-pull-secret-syncer-tvxpw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz","openshift-image-registry/node-ca-tmzvv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal","openshift-multus/multus-6vf2g","openshift-multus/network-metrics-daemon-cfdm7","openshift-network-diagnostics/network-check-target-lx84z","kube-system/konnectivity-agent-gdv2q","kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal"]
Sep 10 19:54:21.316874 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.316851    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.319098 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.319072    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gdv2q"
Sep 10 19:54:21.319672 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.319469    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 19:54:21.319672 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.319480    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 19:54:21.319672 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.319474    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 19:54:21.319831 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.319819    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x8bzk\""
Sep 10 19:54:21.326512 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.326491    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 19:54:21.326647 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.326599    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.326941 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.326925    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cbcc9"
Sep 10 19:54:21.327235 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.327217    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.327996 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.327975    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 19:54:21.328801 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.328787    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dkrpn\""
Sep 10 19:54:21.329335 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.329315    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 19:54:21.329417 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.329381    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 19:54:21.329650 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.329634    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 19:54:21.329750 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.329734    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sr9sk\""
Sep 10 19:54:21.329937 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.329923    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 19:54:21.330965 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.330943    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 19:54:21.331164 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.331147    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 19:54:21.331410 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.331393    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 19:54:21.331734 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.331714    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:21.331808 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:21.331788    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:21.332237 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.332217    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 19:54:21.332729 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.332697    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 19:54:21.332977 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.332822    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 19:54:21.332977 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.332886    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 19:54:21.334597 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.333381    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p8vsm\""
Sep 10 19:54:21.334597 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.333658    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 19:54:21.334597 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.333829    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 19:54:21.334597 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.333984    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 19:54:21.334597 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.334178    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bn7jv\""
Sep 10 19:54:21.334597 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.334342    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 19:54:21.335702 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.335679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4091e39-1788-47af-9032-d55ab85e4089-env-overrides\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.335805 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.335714    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4091e39-1788-47af-9032-d55ab85e4089-ovn-node-metrics-cert\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.335805 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.335739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-sys-fs\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.335805 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.335764    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16811901-cbb8-487a-ba53-94a0227ba869-system-cni-dir\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.335805 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.335789    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/16811901-cbb8-487a-ba53-94a0227ba869-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.336030 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.335816    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-cni-bin\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.336030 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.335832    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-cni-netd\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.336030 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.335875    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c51fb2f9-4a5c-4259-8087-67560d10732b-dbus\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:21.336030 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.335909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.336030 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.335957    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcbt9\" (UniqueName: \"kubernetes.io/projected/16811901-cbb8-487a-ba53-94a0227ba869-kube-api-access-gcbt9\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.336030 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.335991    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-run-systemd\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336035    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-node-log\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-log-socket\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336083    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336107    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qczzh\" (UniqueName: \"kubernetes.io/projected/e4091e39-1788-47af-9032-d55ab85e4089-kube-api-access-qczzh\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8sd2\" (UniqueName: \"kubernetes.io/projected/92385da7-5b0b-4ab6-b9e0-a240595829d4-kube-api-access-b8sd2\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16811901-cbb8-487a-ba53-94a0227ba869-cnibin\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336180    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-kubelet\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336203    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c51fb2f9-4a5c-4259-8087-67560d10732b-kubelet-config\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336228    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16811901-cbb8-487a-ba53-94a0227ba869-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336251    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/16811901-cbb8-487a-ba53-94a0227ba869-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p972p\" (UniqueName: \"kubernetes.io/projected/657fa165-c6ad-4e6a-8e91-867b848fead3-kube-api-access-p972p\") pod \"iptables-alerter-cbcc9\" (UID: \"657fa165-c6ad-4e6a-8e91-867b848fead3\") " pod="openshift-network-operator/iptables-alerter-cbcc9"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-var-lib-openvswitch\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-run-ovn\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.336403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336379    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4091e39-1788-47af-9032-d55ab85e4089-ovnkube-config\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336407    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/85d11857-2f7e-4bae-a7eb-dc6fe5a0b527-konnectivity-ca\") pod \"konnectivity-agent-gdv2q\" (UID: \"85d11857-2f7e-4bae-a7eb-dc6fe5a0b527\") " pod="kube-system/konnectivity-agent-gdv2q"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336473    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/657fa165-c6ad-4e6a-8e91-867b848fead3-iptables-alerter-script\") pod \"iptables-alerter-cbcc9\" (UID: \"657fa165-c6ad-4e6a-8e91-867b848fead3\") " pod="openshift-network-operator/iptables-alerter-cbcc9"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336496    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4091e39-1788-47af-9032-d55ab85e4089-ovnkube-script-lib\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-device-dir\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/85d11857-2f7e-4bae-a7eb-dc6fe5a0b527-agent-certs\") pod \"konnectivity-agent-gdv2q\" (UID: \"85d11857-2f7e-4bae-a7eb-dc6fe5a0b527\") " pod="kube-system/konnectivity-agent-gdv2q"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336619    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-systemd-units\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336643    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-slash\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336676    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-etc-openvswitch\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336701    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-run-openvswitch\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-run-ovn-kubernetes\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336757    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-socket-dir\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336781    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-registration-dir\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16811901-cbb8-487a-ba53-94a0227ba869-cni-binary-copy\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/657fa165-c6ad-4e6a-8e91-867b848fead3-host-slash\") pod \"iptables-alerter-cbcc9\" (UID: \"657fa165-c6ad-4e6a-8e91-867b848fead3\") " pod="openshift-network-operator/iptables-alerter-cbcc9"
Sep 10 19:54:21.337163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.336878    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-run-netns\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.338839 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.338757    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tmzvv"
Sep 10 19:54:21.340618 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.340594    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 19:54:21.342314 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.342190    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 19:54:21.342681 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.342461    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 19:54:21.343592 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.342767    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tjpk5\""
Sep 10 19:54:21.343592 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.342800    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 19:54:21.343592 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.343051    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.346177 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.346155    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tb2sv"
Sep 10 19:54:21.347050 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.347027    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mcln4\""
Sep 10 19:54:21.347139 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.347122    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 19:54:21.347358 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.347337    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 19:54:21.347675 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.347657    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.349000 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.348981    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 19:54:21.349232 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.349214    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 19:54:21.349854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.349835    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2c8q7\""
Sep 10 19:54:21.350121 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.350104    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 19:54:21.350343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.350327    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:21.350416 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:21.350396    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:21.351761 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.351742    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xz2zl\""
Sep 10 19:54:21.352453 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.352353    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:21.352453 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:21.352437    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:21.361577 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.361554    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lsfl2"
Sep 10 19:54:21.373691 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.373661    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lsfl2"
Sep 10 19:54:21.403729 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.403664    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" event={"ID":"10643e86bc3da5fa40e6977a1909b44c","Type":"ContainerStarted","Data":"3eec1dbd306fcaa0868cb4f3ccb32a0c4d383da913a75724663735dca798350b"}
Sep 10 19:54:21.407161 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.407123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" event={"ID":"534d538bb166d56e68f0f739a61e6155","Type":"ContainerStarted","Data":"b91973ed19da46e5a7c3c5156832af2a9ae00531b3dff8881402dab052882954"}
Sep 10 19:54:21.426983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.426952    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 19:54:21.437308 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-var-lib-kubelet\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.437308 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437314    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-multus-conf-dir\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.437495 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-systemd-units\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.437495 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-slash\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.437495 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-run-ovn-kubernetes\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.437495 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437384    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-registration-dir\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.437495 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16811901-cbb8-487a-ba53-94a0227ba869-cni-binary-copy\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.437495 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437397    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-systemd-units\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.437495 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437415    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbn5s\" (UniqueName: \"kubernetes.io/projected/9fce784d-e661-4e90-b91c-49dac8b1e3de-kube-api-access-cbn5s\") pod \"network-metrics-daemon-cfdm7\" (UID: \"9fce784d-e661-4e90-b91c-49dac8b1e3de\") " pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:21.437495 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/657fa165-c6ad-4e6a-8e91-867b848fead3-host-slash\") pod \"iptables-alerter-cbcc9\" (UID: \"657fa165-c6ad-4e6a-8e91-867b848fead3\") " pod="openshift-network-operator/iptables-alerter-cbcc9"
Sep 10 19:54:21.437495 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437462    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-run-netns\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.437495 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4091e39-1788-47af-9032-d55ab85e4089-env-overrides\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.437495 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437484    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-run-ovn-kubernetes\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4091e39-1788-47af-9032-d55ab85e4089-ovn-node-metrics-cert\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/16811901-cbb8-487a-ba53-94a0227ba869-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437541    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-registration-dir\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437558    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-var-lib-cni-bin\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437578    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-var-lib-cni-multus\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437599    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-run-multus-certs\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcbt9\" (UniqueName: \"kubernetes.io/projected/16811901-cbb8-487a-ba53-94a0227ba869-kube-api-access-gcbt9\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437632    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e80f59a-083f-401d-92ba-ae440fb4840e-hosts-file\") pod \"node-resolver-tb2sv\" (UID: \"0e80f59a-083f-401d-92ba-ae440fb4840e\") " pod="openshift-dns/node-resolver-tb2sv"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs\") pod \"network-metrics-daemon-cfdm7\" (UID: \"9fce784d-e661-4e90-b91c-49dac8b1e3de\") " pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-run-systemd\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437687    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-node-log\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437717    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8sd2\" (UniqueName: \"kubernetes.io/projected/92385da7-5b0b-4ab6-b9e0-a240595829d4-kube-api-access-b8sd2\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437731    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-run\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437751    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-sys\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437765    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d6179f7-5886-457f-9b86-a9849a6f3eaa-serviceca\") pod \"node-ca-tmzvv\" (UID: \"2d6179f7-5886-457f-9b86-a9849a6f3eaa\") " pod="openshift-image-registry/node-ca-tmzvv"
Sep 10 19:54:21.437864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437780    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-multus-cni-dir\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437784    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/657fa165-c6ad-4e6a-8e91-867b848fead3-host-slash\") pod \"iptables-alerter-cbcc9\" (UID: \"657fa165-c6ad-4e6a-8e91-867b848fead3\") " pod="openshift-network-operator/iptables-alerter-cbcc9"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d6179f7-5886-457f-9b86-a9849a6f3eaa-host\") pod \"node-ca-tmzvv\" (UID: \"2d6179f7-5886-457f-9b86-a9849a6f3eaa\") " pod="openshift-image-registry/node-ca-tmzvv"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437836    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gf46\" (UniqueName: \"kubernetes.io/projected/2d6179f7-5886-457f-9b86-a9849a6f3eaa-kube-api-access-8gf46\") pod \"node-ca-tmzvv\" (UID: \"2d6179f7-5886-457f-9b86-a9849a6f3eaa\") " pod="openshift-image-registry/node-ca-tmzvv"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437863    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e80f59a-083f-401d-92ba-ae440fb4840e-tmp-dir\") pod \"node-resolver-tb2sv\" (UID: \"0e80f59a-083f-401d-92ba-ae440fb4840e\") " pod="openshift-dns/node-resolver-tb2sv"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csgqd\" (UniqueName: \"kubernetes.io/projected/0e80f59a-083f-401d-92ba-ae440fb4840e-kube-api-access-csgqd\") pod \"node-resolver-tb2sv\" (UID: \"0e80f59a-083f-401d-92ba-ae440fb4840e\") " pod="openshift-dns/node-resolver-tb2sv"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-systemd\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-run-systemd\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437964    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p972p\" (UniqueName: \"kubernetes.io/projected/657fa165-c6ad-4e6a-8e91-867b848fead3-kube-api-access-p972p\") pod \"iptables-alerter-cbcc9\" (UID: \"657fa165-c6ad-4e6a-8e91-867b848fead3\") " pod="openshift-network-operator/iptables-alerter-cbcc9"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-var-lib-openvswitch\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4091e39-1788-47af-9032-d55ab85e4089-ovnkube-config\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438029    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-multus-socket-dir-parent\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-run-netns\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438066    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-etc-kubernetes\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438084    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-device-dir\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438103    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/85d11857-2f7e-4bae-a7eb-dc6fe5a0b527-agent-certs\") pod \"konnectivity-agent-gdv2q\" (UID: \"85d11857-2f7e-4bae-a7eb-dc6fe5a0b527\") " pod="kube-system/konnectivity-agent-gdv2q"
Sep 10 19:54:21.438394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-modprobe-d\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d03870f-dc11-41e0-867d-115382c30a65-tmp\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-system-cni-dir\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438175    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82t8w\" (UniqueName: \"kubernetes.io/projected/45f3fe9c-b6a1-4f28-9630-d3590626af55-kube-api-access-82t8w\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438190    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-etc-openvswitch\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438205    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-run-openvswitch\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-socket-dir\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-host\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-run-netns\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-cnibin\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45f3fe9c-b6a1-4f28-9630-d3590626af55-cni-binary-copy\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-var-lib-kubelet\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45f3fe9c-b6a1-4f28-9630-d3590626af55-multus-daemon-config\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-sys-fs\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438339    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16811901-cbb8-487a-ba53-94a0227ba869-system-cni-dir\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438353    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-sysconfig\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-sysctl-conf\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.439136 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d03870f-dc11-41e0-867d-115382c30a65-etc-tuned\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438404    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5rfn\" (UniqueName: \"kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn\") pod \"network-check-target-lx84z\" (UID: \"a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42\") " pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-cni-bin\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-cni-netd\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c51fb2f9-4a5c-4259-8087-67560d10732b-dbus\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-log-socket\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qczzh\" (UniqueName: \"kubernetes.io/projected/e4091e39-1788-47af-9032-d55ab85e4089-kube-api-access-qczzh\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16811901-cbb8-487a-ba53-94a0227ba869-cnibin\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-kubernetes\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-hostroot\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438591    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-kubelet\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438602    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16811901-cbb8-487a-ba53-94a0227ba869-cni-binary-copy\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438641    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-node-log\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438663    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c51fb2f9-4a5c-4259-8087-67560d10732b-kubelet-config\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438607    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c51fb2f9-4a5c-4259-8087-67560d10732b-kubelet-config\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438674    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438692    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16811901-cbb8-487a-ba53-94a0227ba869-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438724    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4091e39-1788-47af-9032-d55ab85e4089-env-overrides\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438820    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16811901-cbb8-487a-ba53-94a0227ba869-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.438907    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-var-lib-openvswitch\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.439271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4091e39-1788-47af-9032-d55ab85e4089-ovnkube-config\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.439394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-device-dir\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.439401    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.439573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/16811901-cbb8-487a-ba53-94a0227ba869-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.439686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.439805    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16811901-cbb8-487a-ba53-94a0227ba869-system-cni-dir\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.439902    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-etc-openvswitch\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.439938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.439943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-run-openvswitch\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.437463    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-slash\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440036    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-cni-netd\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-log-socket\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c51fb2f9-4a5c-4259-8087-67560d10732b-dbus\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:21.440930 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440096    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-sys-fs\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/16811901-cbb8-487a-ba53-94a0227ba869-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-run-ovn\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-cni-bin\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16811901-cbb8-487a-ba53-94a0227ba869-cnibin\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440306    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/85d11857-2f7e-4bae-a7eb-dc6fe5a0b527-konnectivity-ca\") pod \"konnectivity-agent-gdv2q\" (UID: \"85d11857-2f7e-4bae-a7eb-dc6fe5a0b527\") " pod="kube-system/konnectivity-agent-gdv2q"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92385da7-5b0b-4ab6-b9e0-a240595829d4-socket-dir\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pphvd\" (UniqueName: \"kubernetes.io/projected/4d03870f-dc11-41e0-867d-115382c30a65-kube-api-access-pphvd\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-run-k8s-cni-cncf-io\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/657fa165-c6ad-4e6a-8e91-867b848fead3-iptables-alerter-script\") pod \"iptables-alerter-cbcc9\" (UID: \"657fa165-c6ad-4e6a-8e91-867b848fead3\") " pod="openshift-network-operator/iptables-alerter-cbcc9"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4091e39-1788-47af-9032-d55ab85e4089-ovnkube-script-lib\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-sysctl-d\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440484    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-lib-modules\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:21.440686    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:21.440750    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret podName:c51fb2f9-4a5c-4259-8087-67560d10732b nodeName:}" failed. No retries permitted until 2026-09-10 19:54:21.940730978 +0000 UTC m=+2.991258967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret") pod "global-pull-secret-syncer-tvxpw" (UID: "c51fb2f9-4a5c-4259-8087-67560d10732b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440776    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/85d11857-2f7e-4bae-a7eb-dc6fe5a0b527-konnectivity-ca\") pod \"konnectivity-agent-gdv2q\" (UID: \"85d11857-2f7e-4bae-a7eb-dc6fe5a0b527\") " pod="kube-system/konnectivity-agent-gdv2q"
Sep 10 19:54:21.441836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440906    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4091e39-1788-47af-9032-d55ab85e4089-ovnkube-script-lib\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.442759 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440963    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-host-kubelet\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.442759 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.440967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4091e39-1788-47af-9032-d55ab85e4089-run-ovn\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.442759 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.441206    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/16811901-cbb8-487a-ba53-94a0227ba869-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.442759 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.442087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/657fa165-c6ad-4e6a-8e91-867b848fead3-iptables-alerter-script\") pod \"iptables-alerter-cbcc9\" (UID: \"657fa165-c6ad-4e6a-8e91-867b848fead3\") " pod="openshift-network-operator/iptables-alerter-cbcc9"
Sep 10 19:54:21.444368 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.444346    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4091e39-1788-47af-9032-d55ab85e4089-ovn-node-metrics-cert\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.445009 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.444972    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/85d11857-2f7e-4bae-a7eb-dc6fe5a0b527-agent-certs\") pod \"konnectivity-agent-gdv2q\" (UID: \"85d11857-2f7e-4bae-a7eb-dc6fe5a0b527\") " pod="kube-system/konnectivity-agent-gdv2q"
Sep 10 19:54:21.447091 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.447032    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p972p\" (UniqueName: \"kubernetes.io/projected/657fa165-c6ad-4e6a-8e91-867b848fead3-kube-api-access-p972p\") pod \"iptables-alerter-cbcc9\" (UID: \"657fa165-c6ad-4e6a-8e91-867b848fead3\") " pod="openshift-network-operator/iptables-alerter-cbcc9"
Sep 10 19:54:21.447421 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.447401    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8sd2\" (UniqueName: \"kubernetes.io/projected/92385da7-5b0b-4ab6-b9e0-a240595829d4-kube-api-access-b8sd2\") pod \"aws-ebs-csi-driver-node-n4qpz\" (UID: \"92385da7-5b0b-4ab6-b9e0-a240595829d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.447778 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.447758    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcbt9\" (UniqueName: \"kubernetes.io/projected/16811901-cbb8-487a-ba53-94a0227ba869-kube-api-access-gcbt9\") pod \"multus-additional-cni-plugins-6n4sz\" (UID: \"16811901-cbb8-487a-ba53-94a0227ba869\") " pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.448680 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.448639    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qczzh\" (UniqueName: \"kubernetes.io/projected/e4091e39-1788-47af-9032-d55ab85e4089-kube-api-access-qczzh\") pod \"ovnkube-node-xstzr\" (UID: \"e4091e39-1788-47af-9032-d55ab85e4089\") " pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.471503 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.471474    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:54:21.540772 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.540741    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gf46\" (UniqueName: \"kubernetes.io/projected/2d6179f7-5886-457f-9b86-a9849a6f3eaa-kube-api-access-8gf46\") pod \"node-ca-tmzvv\" (UID: \"2d6179f7-5886-457f-9b86-a9849a6f3eaa\") " pod="openshift-image-registry/node-ca-tmzvv"
Sep 10 19:54:21.540969 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.540778    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e80f59a-083f-401d-92ba-ae440fb4840e-tmp-dir\") pod \"node-resolver-tb2sv\" (UID: \"0e80f59a-083f-401d-92ba-ae440fb4840e\") " pod="openshift-dns/node-resolver-tb2sv"
Sep 10 19:54:21.540969 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.540803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csgqd\" (UniqueName: \"kubernetes.io/projected/0e80f59a-083f-401d-92ba-ae440fb4840e-kube-api-access-csgqd\") pod \"node-resolver-tb2sv\" (UID: \"0e80f59a-083f-401d-92ba-ae440fb4840e\") " pod="openshift-dns/node-resolver-tb2sv"
Sep 10 19:54:21.540969 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.540827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-systemd\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.540969 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.540865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-multus-socket-dir-parent\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.540969 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.540887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-run-netns\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.540969 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.540910    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-etc-kubernetes\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.540969 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.540955    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-etc-kubernetes\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.540969 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.540962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-systemd\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.540970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-multus-socket-dir-parent\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541009    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-run-netns\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-modprobe-d\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d03870f-dc11-41e0-867d-115382c30a65-tmp\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541122    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-modprobe-d\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541127    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-system-cni-dir\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541132    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e80f59a-083f-401d-92ba-ae440fb4840e-tmp-dir\") pod \"node-resolver-tb2sv\" (UID: \"0e80f59a-083f-401d-92ba-ae440fb4840e\") " pod="openshift-dns/node-resolver-tb2sv"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541154    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82t8w\" (UniqueName: \"kubernetes.io/projected/45f3fe9c-b6a1-4f28-9630-d3590626af55-kube-api-access-82t8w\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-host\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-cnibin\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541281    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-system-cni-dir\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45f3fe9c-b6a1-4f28-9630-d3590626af55-cni-binary-copy\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-var-lib-kubelet\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45f3fe9c-b6a1-4f28-9630-d3590626af55-multus-daemon-config\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.541343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-host\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541371    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-sysconfig\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541394    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-sysctl-conf\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-var-lib-kubelet\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d03870f-dc11-41e0-867d-115382c30a65-etc-tuned\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541457    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rfn\" (UniqueName: \"kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn\") pod \"network-check-target-lx84z\" (UID: \"a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42\") " pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541470    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-cnibin\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541490    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-kubernetes\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-sysconfig\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541517    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-hostroot\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-hostroot\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphvd\" (UniqueName: \"kubernetes.io/projected/4d03870f-dc11-41e0-867d-115382c30a65-kube-api-access-pphvd\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541633    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-run-k8s-cni-cncf-io\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-sysctl-d\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-lib-modules\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-var-lib-kubelet\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541726    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-multus-conf-dir\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541755    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbn5s\" (UniqueName: \"kubernetes.io/projected/9fce784d-e661-4e90-b91c-49dac8b1e3de-kube-api-access-cbn5s\") pod \"network-metrics-daemon-cfdm7\" (UID: \"9fce784d-e661-4e90-b91c-49dac8b1e3de\") " pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:21.542107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-var-lib-cni-bin\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541807    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-var-lib-cni-multus\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541832    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-run-multus-certs\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e80f59a-083f-401d-92ba-ae440fb4840e-hosts-file\") pod \"node-resolver-tb2sv\" (UID: \"0e80f59a-083f-401d-92ba-ae440fb4840e\") " pod="openshift-dns/node-resolver-tb2sv"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541886    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs\") pod \"network-metrics-daemon-cfdm7\" (UID: \"9fce784d-e661-4e90-b91c-49dac8b1e3de\") " pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-run\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45f3fe9c-b6a1-4f28-9630-d3590626af55-multus-daemon-config\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541954    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-sys\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541955    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45f3fe9c-b6a1-4f28-9630-d3590626af55-cni-binary-copy\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541965    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-sysctl-conf\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.541978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d6179f7-5886-457f-9b86-a9849a6f3eaa-serviceca\") pod \"node-ca-tmzvv\" (UID: \"2d6179f7-5886-457f-9b86-a9849a6f3eaa\") " pod="openshift-image-registry/node-ca-tmzvv"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-multus-cni-dir\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542032    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-run-k8s-cni-cncf-io\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d6179f7-5886-457f-9b86-a9849a6f3eaa-host\") pod \"node-ca-tmzvv\" (UID: \"2d6179f7-5886-457f-9b86-a9849a6f3eaa\") " pod="openshift-image-registry/node-ca-tmzvv"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542077    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e80f59a-083f-401d-92ba-ae440fb4840e-hosts-file\") pod \"node-resolver-tb2sv\" (UID: \"0e80f59a-083f-401d-92ba-ae440fb4840e\") " pod="openshift-dns/node-resolver-tb2sv"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542122    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-kubernetes\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542159    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-var-lib-cni-bin\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:21.542179    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542195    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-var-lib-cni-multus\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.542861 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542206    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d6179f7-5886-457f-9b86-a9849a6f3eaa-host\") pod \"node-ca-tmzvv\" (UID: \"2d6179f7-5886-457f-9b86-a9849a6f3eaa\") " pod="openshift-image-registry/node-ca-tmzvv"
Sep 10 19:54:21.543519 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542225    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-multus-conf-dir\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.543519 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:21.542242    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs podName:9fce784d-e661-4e90-b91c-49dac8b1e3de nodeName:}" failed. No retries permitted until 2026-09-10 19:54:22.042226238 +0000 UTC m=+3.092754224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs") pod "network-metrics-daemon-cfdm7" (UID: "9fce784d-e661-4e90-b91c-49dac8b1e3de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:54:21.543519 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-multus-cni-dir\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.543519 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542314    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-sys\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.543519 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542346    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-lib-modules\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.543519 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-run\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.543519 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45f3fe9c-b6a1-4f28-9630-d3590626af55-host-run-multus-certs\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.543519 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542412    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-var-lib-kubelet\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.543519 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542426    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d03870f-dc11-41e0-867d-115382c30a65-etc-sysctl-d\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.543519 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.542485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d6179f7-5886-457f-9b86-a9849a6f3eaa-serviceca\") pod \"node-ca-tmzvv\" (UID: \"2d6179f7-5886-457f-9b86-a9849a6f3eaa\") " pod="openshift-image-registry/node-ca-tmzvv"
Sep 10 19:54:21.544058 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.544037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d03870f-dc11-41e0-867d-115382c30a65-tmp\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.544606 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.544586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d03870f-dc11-41e0-867d-115382c30a65-etc-tuned\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.548153 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:21.548129    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:54:21.548258 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:21.548159    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:54:21.548258 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:21.548174    2561 projected.go:196] Error preparing data for projected volume kube-api-access-t5rfn for pod openshift-network-diagnostics/network-check-target-lx84z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:54:21.548258 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:21.548237    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn podName:a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42 nodeName:}" failed. No retries permitted until 2026-09-10 19:54:22.048219027 +0000 UTC m=+3.098747008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t5rfn" (UniqueName: "kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn") pod "network-check-target-lx84z" (UID: "a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:54:21.549073 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.549053    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gf46\" (UniqueName: \"kubernetes.io/projected/2d6179f7-5886-457f-9b86-a9849a6f3eaa-kube-api-access-8gf46\") pod \"node-ca-tmzvv\" (UID: \"2d6179f7-5886-457f-9b86-a9849a6f3eaa\") " pod="openshift-image-registry/node-ca-tmzvv"
Sep 10 19:54:21.549164 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.549095    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csgqd\" (UniqueName: \"kubernetes.io/projected/0e80f59a-083f-401d-92ba-ae440fb4840e-kube-api-access-csgqd\") pod \"node-resolver-tb2sv\" (UID: \"0e80f59a-083f-401d-92ba-ae440fb4840e\") " pod="openshift-dns/node-resolver-tb2sv"
Sep 10 19:54:21.549380 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.549336    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82t8w\" (UniqueName: \"kubernetes.io/projected/45f3fe9c-b6a1-4f28-9630-d3590626af55-kube-api-access-82t8w\") pod \"multus-6vf2g\" (UID: \"45f3fe9c-b6a1-4f28-9630-d3590626af55\") " pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.550418 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.550397    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphvd\" (UniqueName: \"kubernetes.io/projected/4d03870f-dc11-41e0-867d-115382c30a65-kube-api-access-pphvd\") pod \"tuned-99z2f\" (UID: \"4d03870f-dc11-41e0-867d-115382c30a65\") " pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.550840 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.550823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbn5s\" (UniqueName: \"kubernetes.io/projected/9fce784d-e661-4e90-b91c-49dac8b1e3de-kube-api-access-cbn5s\") pod \"network-metrics-daemon-cfdm7\" (UID: \"9fce784d-e661-4e90-b91c-49dac8b1e3de\") " pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:21.641426 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.641330    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz"
Sep 10 19:54:21.649690 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.649652    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gdv2q"
Sep 10 19:54:21.664488 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.664462    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6n4sz"
Sep 10 19:54:21.670377 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.670355    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cbcc9"
Sep 10 19:54:21.679182 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.679156    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:21.689869 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.689846    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tmzvv"
Sep 10 19:54:21.698698 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.698679    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-99z2f"
Sep 10 19:54:21.707449 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.707426    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tb2sv"
Sep 10 19:54:21.716277 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.716252    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6vf2g"
Sep 10 19:54:21.815830 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.815797    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:54:21.944659 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:21.944574    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:21.945051 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:21.945010    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:54:21.945206 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:21.945110    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret podName:c51fb2f9-4a5c-4259-8087-67560d10732b nodeName:}" failed. No retries permitted until 2026-09-10 19:54:22.945072781 +0000 UTC m=+3.995600762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret") pod "global-pull-secret-syncer-tvxpw" (UID: "c51fb2f9-4a5c-4259-8087-67560d10732b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:54:22.045708 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:22.045668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs\") pod \"network-metrics-daemon-cfdm7\" (UID: \"9fce784d-e661-4e90-b91c-49dac8b1e3de\") " pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:22.045885 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:22.045838    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:54:22.045937 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:22.045910    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs podName:9fce784d-e661-4e90-b91c-49dac8b1e3de nodeName:}" failed. No retries permitted until 2026-09-10 19:54:23.045895862 +0000 UTC m=+4.096423841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs") pod "network-metrics-daemon-cfdm7" (UID: "9fce784d-e661-4e90-b91c-49dac8b1e3de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:54:22.147070 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:22.147028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rfn\" (UniqueName: \"kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn\") pod \"network-check-target-lx84z\" (UID: \"a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42\") " pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:22.147244 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:22.147226    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:54:22.147304 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:22.147255    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:54:22.147304 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:22.147269    2561 projected.go:196] Error preparing data for projected volume kube-api-access-t5rfn for pod openshift-network-diagnostics/network-check-target-lx84z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:54:22.147397 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:22.147358    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn podName:a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42 nodeName:}" failed. No retries permitted until 2026-09-10 19:54:23.147336633 +0000 UTC m=+4.197864620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t5rfn" (UniqueName: "kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn") pod "network-check-target-lx84z" (UID: "a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:54:22.380710 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:22.380599    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 19:49:21 +0000 UTC" deadline="2028-03-25 06:38:36.086047905 +0000 UTC"
Sep 10 19:54:22.380710 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:22.380655    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13474h44m13.705396497s"
Sep 10 19:54:22.398043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:22.398011    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:22.398211 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:22.398139    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:22.456502 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:22.456329    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657fa165_c6ad_4e6a_8e91_867b848fead3.slice/crio-576933ff8121777e7247e959cda34c5ef4d7bb7fe2b136aa5785db5eb3663f38 WatchSource:0}: Error finding container 576933ff8121777e7247e959cda34c5ef4d7bb7fe2b136aa5785db5eb3663f38: Status 404 returned error can't find the container with id 576933ff8121777e7247e959cda34c5ef4d7bb7fe2b136aa5785db5eb3663f38
Sep 10 19:54:22.457546 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:22.457505    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d11857_2f7e_4bae_a7eb_dc6fe5a0b527.slice/crio-1529c5c5b4a890cc963973dc933040f7272c0dc2e4a90665d6c4b391d57c3bae WatchSource:0}: Error finding container 1529c5c5b4a890cc963973dc933040f7272c0dc2e4a90665d6c4b391d57c3bae: Status 404 returned error can't find the container with id 1529c5c5b4a890cc963973dc933040f7272c0dc2e4a90665d6c4b391d57c3bae
Sep 10 19:54:22.458363 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:22.458225    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6179f7_5886_457f_9b86_a9849a6f3eaa.slice/crio-f9bfc17a00262f72528bcfb3eb7d10f0758df5cdd61cba8ca4617271ee52e1d1 WatchSource:0}: Error finding container f9bfc17a00262f72528bcfb3eb7d10f0758df5cdd61cba8ca4617271ee52e1d1: Status 404 returned error can't find the container with id f9bfc17a00262f72528bcfb3eb7d10f0758df5cdd61cba8ca4617271ee52e1d1
Sep 10 19:54:22.459468 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:22.459438    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f3fe9c_b6a1_4f28_9630_d3590626af55.slice/crio-cdbd376a75fd896010adefedfa2cd1f78f9a5a6737d54986111cc301c14283be WatchSource:0}: Error finding container cdbd376a75fd896010adefedfa2cd1f78f9a5a6737d54986111cc301c14283be: Status 404 returned error can't find the container with id cdbd376a75fd896010adefedfa2cd1f78f9a5a6737d54986111cc301c14283be
Sep 10 19:54:22.461286 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:22.460857    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e80f59a_083f_401d_92ba_ae440fb4840e.slice/crio-69052028d4998f0d4f3d0a00cbae01de9b72f9dacdda601d5a39f291e1ebf21c WatchSource:0}: Error finding container 69052028d4998f0d4f3d0a00cbae01de9b72f9dacdda601d5a39f291e1ebf21c: Status 404 returned error can't find the container with id 69052028d4998f0d4f3d0a00cbae01de9b72f9dacdda601d5a39f291e1ebf21c
Sep 10 19:54:22.462578 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:22.462558    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d03870f_dc11_41e0_867d_115382c30a65.slice/crio-11b5082982a282169eaf2185c81c2e469f1af4fad430a62c8ea67ea56876f12b WatchSource:0}: Error finding container 11b5082982a282169eaf2185c81c2e469f1af4fad430a62c8ea67ea56876f12b: Status 404 returned error can't find the container with id 11b5082982a282169eaf2185c81c2e469f1af4fad430a62c8ea67ea56876f12b
Sep 10 19:54:22.484439 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:22.484412    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92385da7_5b0b_4ab6_b9e0_a240595829d4.slice/crio-7b60e4f9e72c4b3c47933758760f9ea3a852bc169a0cd148b6c325953c9a3e85 WatchSource:0}: Error finding container 7b60e4f9e72c4b3c47933758760f9ea3a852bc169a0cd148b6c325953c9a3e85: Status 404 returned error can't find the container with id 7b60e4f9e72c4b3c47933758760f9ea3a852bc169a0cd148b6c325953c9a3e85
Sep 10 19:54:22.485291 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:22.485259    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16811901_cbb8_487a_ba53_94a0227ba869.slice/crio-3fd335658827cbedb00ed415ebc145b9b86431788e7e063eeafd8ff0d8d82688 WatchSource:0}: Error finding container 3fd335658827cbedb00ed415ebc145b9b86431788e7e063eeafd8ff0d8d82688: Status 404 returned error can't find the container with id 3fd335658827cbedb00ed415ebc145b9b86431788e7e063eeafd8ff0d8d82688
Sep 10 19:54:22.486246 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:22.486201    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4091e39_1788_47af_9032_d55ab85e4089.slice/crio-147dedec7e9b1f3d8a5b4bcc583eb42d5dbedd7b708ba362dc1805c14e00459a WatchSource:0}: Error finding container 147dedec7e9b1f3d8a5b4bcc583eb42d5dbedd7b708ba362dc1805c14e00459a: Status 404 returned error can't find the container with id 147dedec7e9b1f3d8a5b4bcc583eb42d5dbedd7b708ba362dc1805c14e00459a
Sep 10 19:54:22.953372 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:22.953326    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:22.953571 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:22.953482    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:54:22.953571 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:22.953567    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret podName:c51fb2f9-4a5c-4259-8087-67560d10732b nodeName:}" failed. No retries permitted until 2026-09-10 19:54:24.953546035 +0000 UTC m=+6.004074023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret") pod "global-pull-secret-syncer-tvxpw" (UID: "c51fb2f9-4a5c-4259-8087-67560d10732b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:54:23.054184 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.053720    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs\") pod \"network-metrics-daemon-cfdm7\" (UID: \"9fce784d-e661-4e90-b91c-49dac8b1e3de\") " pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:23.054184 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:23.053825    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:54:23.054184 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:23.053873    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs podName:9fce784d-e661-4e90-b91c-49dac8b1e3de nodeName:}" failed. No retries permitted until 2026-09-10 19:54:25.053859811 +0000 UTC m=+6.104387791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs") pod "network-metrics-daemon-cfdm7" (UID: "9fce784d-e661-4e90-b91c-49dac8b1e3de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:54:23.154619 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.154579    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rfn\" (UniqueName: \"kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn\") pod \"network-check-target-lx84z\" (UID: \"a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42\") " pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:23.154844 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:23.154804    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:54:23.154844 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:23.154826    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:54:23.154844 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:23.154839    2561 projected.go:196] Error preparing data for projected volume kube-api-access-t5rfn for pod openshift-network-diagnostics/network-check-target-lx84z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:54:23.154991 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:23.154901    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn podName:a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42 nodeName:}" failed. No retries permitted until 2026-09-10 19:54:25.154882946 +0000 UTC m=+6.205410937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t5rfn" (UniqueName: "kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn") pod "network-check-target-lx84z" (UID: "a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:54:23.382202 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.382108    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 19:49:21 +0000 UTC" deadline="2028-05-05 07:18:08.635946706 +0000 UTC"
Sep 10 19:54:23.382202 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.382152    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14459h23m45.253800099s"
Sep 10 19:54:23.398699 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.398666    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:23.398849 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:23.398799    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:23.399480 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.399455    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:23.399603 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:23.399581    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:23.420959 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.420873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gdv2q" event={"ID":"85d11857-2f7e-4bae-a7eb-dc6fe5a0b527","Type":"ContainerStarted","Data":"1529c5c5b4a890cc963973dc933040f7272c0dc2e4a90665d6c4b391d57c3bae"}
Sep 10 19:54:23.427310 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.427272    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-99z2f" event={"ID":"4d03870f-dc11-41e0-867d-115382c30a65","Type":"ContainerStarted","Data":"11b5082982a282169eaf2185c81c2e469f1af4fad430a62c8ea67ea56876f12b"}
Sep 10 19:54:23.430018 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.429953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tb2sv" event={"ID":"0e80f59a-083f-401d-92ba-ae440fb4840e","Type":"ContainerStarted","Data":"69052028d4998f0d4f3d0a00cbae01de9b72f9dacdda601d5a39f291e1ebf21c"}
Sep 10 19:54:23.432123 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.432091    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6vf2g" event={"ID":"45f3fe9c-b6a1-4f28-9630-d3590626af55","Type":"ContainerStarted","Data":"cdbd376a75fd896010adefedfa2cd1f78f9a5a6737d54986111cc301c14283be"}
Sep 10 19:54:23.440553 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.440442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cbcc9" event={"ID":"657fa165-c6ad-4e6a-8e91-867b848fead3","Type":"ContainerStarted","Data":"576933ff8121777e7247e959cda34c5ef4d7bb7fe2b136aa5785db5eb3663f38"}
Sep 10 19:54:23.452350 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.452297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" event={"ID":"10643e86bc3da5fa40e6977a1909b44c","Type":"ContainerStarted","Data":"b4681579e5daee083f80e576718f10ddc3f46932d4361e567be26146561c1b8c"}
Sep 10 19:54:23.468221 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.466837    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr" event={"ID":"e4091e39-1788-47af-9032-d55ab85e4089","Type":"ContainerStarted","Data":"147dedec7e9b1f3d8a5b4bcc583eb42d5dbedd7b708ba362dc1805c14e00459a"}
Sep 10 19:54:23.468221 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.467957    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" podStartSLOduration=3.467936709 podStartE2EDuration="3.467936709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:54:23.466937331 +0000 UTC m=+4.517465323" watchObservedRunningTime="2026-09-10 19:54:23.467936709 +0000 UTC m=+4.518464700"
Sep 10 19:54:23.480313 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.480266    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz" event={"ID":"92385da7-5b0b-4ab6-b9e0-a240595829d4","Type":"ContainerStarted","Data":"7b60e4f9e72c4b3c47933758760f9ea3a852bc169a0cd148b6c325953c9a3e85"}
Sep 10 19:54:23.488634 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.488575    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tmzvv" event={"ID":"2d6179f7-5886-457f-9b86-a9849a6f3eaa","Type":"ContainerStarted","Data":"f9bfc17a00262f72528bcfb3eb7d10f0758df5cdd61cba8ca4617271ee52e1d1"}
Sep 10 19:54:23.500423 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.499546    2561 generic.go:356] "Generic (PLEG): container finished" podID="534d538bb166d56e68f0f739a61e6155" containerID="c6037f2baeb0736fa257f25aa6da1961a061c0675bd40deb6d800a13887821a6" exitCode=0
Sep 10 19:54:23.500423 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.499636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" event={"ID":"534d538bb166d56e68f0f739a61e6155","Type":"ContainerDied","Data":"c6037f2baeb0736fa257f25aa6da1961a061c0675bd40deb6d800a13887821a6"}
Sep 10 19:54:23.516461 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:23.516423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6n4sz" event={"ID":"16811901-cbb8-487a-ba53-94a0227ba869","Type":"ContainerStarted","Data":"3fd335658827cbedb00ed415ebc145b9b86431788e7e063eeafd8ff0d8d82688"}
Sep 10 19:54:24.397926 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:24.397874    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:24.398553 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:24.398006    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:24.538634 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:24.538595    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" event={"ID":"534d538bb166d56e68f0f739a61e6155","Type":"ContainerStarted","Data":"d5a88b64910f4f2541b0db20403eb8aa9b16bb31ab4e88805913867410f29d7a"}
Sep 10 19:54:24.971016 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:24.970947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:24.971222 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:24.971181    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:54:24.971291 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:24.971248    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret podName:c51fb2f9-4a5c-4259-8087-67560d10732b nodeName:}" failed. No retries permitted until 2026-09-10 19:54:28.971229787 +0000 UTC m=+10.021757774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret") pod "global-pull-secret-syncer-tvxpw" (UID: "c51fb2f9-4a5c-4259-8087-67560d10732b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:54:25.072442 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:25.072398    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs\") pod \"network-metrics-daemon-cfdm7\" (UID: \"9fce784d-e661-4e90-b91c-49dac8b1e3de\") " pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:25.072642 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:25.072516    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:54:25.072642 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:25.072614    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs podName:9fce784d-e661-4e90-b91c-49dac8b1e3de nodeName:}" failed. No retries permitted until 2026-09-10 19:54:29.072592125 +0000 UTC m=+10.123120115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs") pod "network-metrics-daemon-cfdm7" (UID: "9fce784d-e661-4e90-b91c-49dac8b1e3de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:54:25.174255 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:25.173625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rfn\" (UniqueName: \"kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn\") pod \"network-check-target-lx84z\" (UID: \"a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42\") " pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:25.174255 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:25.173784    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:54:25.174255 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:25.173807    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:54:25.174255 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:25.173820    2561 projected.go:196] Error preparing data for projected volume kube-api-access-t5rfn for pod openshift-network-diagnostics/network-check-target-lx84z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:54:25.174255 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:25.173882    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn podName:a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42 nodeName:}" failed. No retries permitted until 2026-09-10 19:54:29.173864187 +0000 UTC m=+10.224392162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t5rfn" (UniqueName: "kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn") pod "network-check-target-lx84z" (UID: "a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:54:25.399183 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:25.398643    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:25.399183 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:25.398775    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:25.399979 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:25.399783    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:25.399979 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:25.399890    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:26.397675 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:26.397638    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:26.397870 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:26.397785    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:27.399854 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:27.399822    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:27.400341 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:27.399932    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:27.400341 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:27.399987    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:27.400341 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:27.400056    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:28.398131 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:28.398052    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:28.398296 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:28.398185    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:29.005782 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:29.005732    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:29.006205 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:29.005937    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:54:29.006205 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:29.005996    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret podName:c51fb2f9-4a5c-4259-8087-67560d10732b nodeName:}" failed. No retries permitted until 2026-09-10 19:54:37.005979244 +0000 UTC m=+18.056507228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret") pod "global-pull-secret-syncer-tvxpw" (UID: "c51fb2f9-4a5c-4259-8087-67560d10732b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:54:29.106543 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:29.106488    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs\") pod \"network-metrics-daemon-cfdm7\" (UID: \"9fce784d-e661-4e90-b91c-49dac8b1e3de\") " pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:29.106722 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:29.106664    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:54:29.106787 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:29.106729    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs podName:9fce784d-e661-4e90-b91c-49dac8b1e3de nodeName:}" failed. No retries permitted until 2026-09-10 19:54:37.106710631 +0000 UTC m=+18.157238604 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs") pod "network-metrics-daemon-cfdm7" (UID: "9fce784d-e661-4e90-b91c-49dac8b1e3de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:54:29.206969 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:29.206935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rfn\" (UniqueName: \"kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn\") pod \"network-check-target-lx84z\" (UID: \"a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42\") " pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:29.207131 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:29.207093    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:54:29.207131 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:29.207115    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:54:29.207131 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:29.207127    2561 projected.go:196] Error preparing data for projected volume kube-api-access-t5rfn for pod openshift-network-diagnostics/network-check-target-lx84z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:54:29.207323 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:29.207183    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn podName:a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42 nodeName:}" failed. No retries permitted until 2026-09-10 19:54:37.207166963 +0000 UTC m=+18.257694930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t5rfn" (UniqueName: "kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn") pod "network-check-target-lx84z" (UID: "a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:54:29.398692 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:29.398627    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:29.398813 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:29.398702    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:29.398813 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:29.398709    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:29.398813 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:29.398761    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:30.398485 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:30.398442    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:30.398957 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:30.398616    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:31.399517 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:31.399480    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:31.399517 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:31.399511    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:31.400034 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:31.399623    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:31.400034 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:31.399744    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:32.398382 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:32.398344    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:32.398543 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:32.398470    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:33.398256 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:33.397978    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:33.398725 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:33.398300    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:33.398725 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:33.397979    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:33.398882 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:33.398730    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:34.398076 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:34.398036    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:34.398269 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:34.398155    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:35.398768 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:35.398260    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:35.398768 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:35.398380    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:35.398768 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:35.398523    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:35.398768 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:35.398653    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:36.397983 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:36.397947    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:36.398196 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:36.398057    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:37.066908 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:37.066817    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:37.067352 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:37.066944    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:54:37.067352 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:37.067007    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret podName:c51fb2f9-4a5c-4259-8087-67560d10732b nodeName:}" failed. No retries permitted until 2026-09-10 19:54:53.06699215 +0000 UTC m=+34.117520121 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret") pod "global-pull-secret-syncer-tvxpw" (UID: "c51fb2f9-4a5c-4259-8087-67560d10732b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:54:37.167883 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:37.167832    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs\") pod \"network-metrics-daemon-cfdm7\" (UID: \"9fce784d-e661-4e90-b91c-49dac8b1e3de\") " pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:37.168072 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:37.168001    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:54:37.168125 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:37.168076    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs podName:9fce784d-e661-4e90-b91c-49dac8b1e3de nodeName:}" failed. No retries permitted until 2026-09-10 19:54:53.168053074 +0000 UTC m=+34.218581056 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs") pod "network-metrics-daemon-cfdm7" (UID: "9fce784d-e661-4e90-b91c-49dac8b1e3de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:54:37.269052 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:37.269015    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rfn\" (UniqueName: \"kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn\") pod \"network-check-target-lx84z\" (UID: \"a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42\") " pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:37.269228 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:37.269171    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:54:37.269228 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:37.269189    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:54:37.269228 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:37.269198    2561 projected.go:196] Error preparing data for projected volume kube-api-access-t5rfn for pod openshift-network-diagnostics/network-check-target-lx84z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:54:37.269358 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:37.269257    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn podName:a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42 nodeName:}" failed. No retries permitted until 2026-09-10 19:54:53.269241911 +0000 UTC m=+34.319769883 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t5rfn" (UniqueName: "kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn") pod "network-check-target-lx84z" (UID: "a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:54:37.398269 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:37.398175    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:37.398429 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:37.398326    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:37.398429 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:37.398376    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:37.398565 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:37.398497    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:38.397773 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:38.397741    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:38.398241 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:38.397867    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:39.398776 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:39.398740    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:39.399236 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:39.398855    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:39.399236 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:39.398900    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:39.399236 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:39.399000    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:40.397703 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:40.397671    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:40.397887 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:40.397777    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:41.398746 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.398494    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:41.399402 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.398521    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:41.399402 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:41.398860    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:41.399402 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:41.398920    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:41.578214 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.578176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr" event={"ID":"e4091e39-1788-47af-9032-d55ab85e4089","Type":"ContainerStarted","Data":"c05254ad2f898d001132f3827d7c51d8d9dc06fa80819d98d6542fe23584530d"}
Sep 10 19:54:41.578400 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.578266    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr" event={"ID":"e4091e39-1788-47af-9032-d55ab85e4089","Type":"ContainerStarted","Data":"8e4470a8790d381a733e0c76b236d0e3cd00b1b6f21791dec6621a576edce3c1"}
Sep 10 19:54:41.578400 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.578345    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr" event={"ID":"e4091e39-1788-47af-9032-d55ab85e4089","Type":"ContainerStarted","Data":"2582ac1c4c461c17a1f0859456c8cf732588f5f556e7bff36481af91184224e2"}
Sep 10 19:54:41.578400 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.578396    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr" event={"ID":"e4091e39-1788-47af-9032-d55ab85e4089","Type":"ContainerStarted","Data":"31fab89b241924f25338f50f771876f4ded8970341882a94b6048a58ccb22eb7"}
Sep 10 19:54:41.578570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.578442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr" event={"ID":"e4091e39-1788-47af-9032-d55ab85e4089","Type":"ContainerStarted","Data":"79d1a75ef722cf50320209f0173a07fdeab8c830deeb666b4ee6ee662bcb0fed"}
Sep 10 19:54:41.578570 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.578472    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr" event={"ID":"e4091e39-1788-47af-9032-d55ab85e4089","Type":"ContainerStarted","Data":"f37915859a1017ec42d659d495b769d72eea960707e86ce394c5609858403ee7"}
Sep 10 19:54:41.579796 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.579774    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz" event={"ID":"92385da7-5b0b-4ab6-b9e0-a240595829d4","Type":"ContainerStarted","Data":"b874be851b61a8784825d364d29e17a080395d684ec6be9159cd86e30dc56ed7"}
Sep 10 19:54:41.581134 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.581117    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tmzvv" event={"ID":"2d6179f7-5886-457f-9b86-a9849a6f3eaa","Type":"ContainerStarted","Data":"9b3bbd2cd8d5e4625d898ee2a9e21ac4b448703ef771fd61895cb2c2b6ed328f"}
Sep 10 19:54:41.582578 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.582552    2561 generic.go:356] "Generic (PLEG): container finished" podID="16811901-cbb8-487a-ba53-94a0227ba869" containerID="be51ed6d90ede701a91891ed0ecab03ecea4dd030848c1bcc1e28bdb8234347f" exitCode=0
Sep 10 19:54:41.582680 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.582633    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6n4sz" event={"ID":"16811901-cbb8-487a-ba53-94a0227ba869","Type":"ContainerDied","Data":"be51ed6d90ede701a91891ed0ecab03ecea4dd030848c1bcc1e28bdb8234347f"}
Sep 10 19:54:41.584086 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.584065    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gdv2q" event={"ID":"85d11857-2f7e-4bae-a7eb-dc6fe5a0b527","Type":"ContainerStarted","Data":"0e1444e3d80c58531e93b5af78c838c4849e9b9eeb88d896ded7435ed6cfc68e"}
Sep 10 19:54:41.585485 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.585463    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-99z2f" event={"ID":"4d03870f-dc11-41e0-867d-115382c30a65","Type":"ContainerStarted","Data":"71a63d081e7ccf10358520914d045eab3b6f9e1ec684638251e22490cf3ee6e7"}
Sep 10 19:54:41.587051 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.587018    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tb2sv" event={"ID":"0e80f59a-083f-401d-92ba-ae440fb4840e","Type":"ContainerStarted","Data":"02278d87168942a1d39b0f0fd6f37753da5d87d984985b7b82b3fa7b517caa1e"}
Sep 10 19:54:41.588515 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.588493    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6vf2g" event={"ID":"45f3fe9c-b6a1-4f28-9630-d3590626af55","Type":"ContainerStarted","Data":"ac26d94e828483dbb881d26b8a1ead4b6c954050ebf793a7811b9437c8a26cc7"}
Sep 10 19:54:41.597831 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.597792    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tmzvv" podStartSLOduration=4.554952701 podStartE2EDuration="22.597780707s" totalImagesPullingTime="18.042828006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:54:41.597190045 +0000 UTC m=+22.647718034" watchObservedRunningTime="2026-09-10 19:54:41.597780707 +0000 UTC m=+22.648308695"
Sep 10 19:54:41.597936 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.597863    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" podStartSLOduration=21.59785864 podStartE2EDuration="21.59785864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:54:24.554748061 +0000 UTC m=+5.605276051" watchObservedRunningTime="2026-09-10 19:54:41.59785864 +0000 UTC m=+22.648386630"
Sep 10 19:54:41.615914 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.615863    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6vf2g" podStartSLOduration=4.374341253 podStartE2EDuration="22.615845455s" totalImagesPullingTime="18.241504202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:54:41.615224586 +0000 UTC m=+22.665752576" watchObservedRunningTime="2026-09-10 19:54:41.615845455 +0000 UTC m=+22.666373448"
Sep 10 19:54:41.638010 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.637953    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-99z2f" podStartSLOduration=4.6147108 podStartE2EDuration="22.637936261s" totalImagesPullingTime="18.023225461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:54:41.637922576 +0000 UTC m=+22.688450566" watchObservedRunningTime="2026-09-10 19:54:41.637936261 +0000 UTC m=+22.688464250"
Sep 10 19:54:41.655443 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.655381    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tb2sv" podStartSLOduration=4.6344204 podStartE2EDuration="22.655365768s" totalImagesPullingTime="18.020945368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:54:41.655311855 +0000 UTC m=+22.705839845" watchObservedRunningTime="2026-09-10 19:54:41.655365768 +0000 UTC m=+22.705893757"
Sep 10 19:54:41.671403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.671342    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gdv2q" podStartSLOduration=8.417225032 podStartE2EDuration="22.671324982s" totalImagesPullingTime="14.25409995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:54:41.670873371 +0000 UTC m=+22.721401359" watchObservedRunningTime="2026-09-10 19:54:41.671324982 +0000 UTC m=+22.721852971"
Sep 10 19:54:41.744952 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.744911    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:41.860642 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:41.860605    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:42.139881 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:42.139852    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:42.269331 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:42.269298    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 19:54:42.398041 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:42.397966    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:42.398184 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:42.398115    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:42.405301 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:42.405199    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T19:54:42.269315648Z","UUID":"f2f7f10a-a1cf-49d8-97c7-20199ecca676","Handler":null,"Name":"","Endpoint":""}
Sep 10 19:54:42.406951 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:42.406933    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 19:54:42.407086 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:42.406970    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 19:54:42.593787 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:42.593740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz" event={"ID":"92385da7-5b0b-4ab6-b9e0-a240595829d4","Type":"ContainerStarted","Data":"651f9be5b5c755ec43e570f8b9ca233fd1845622d2902a1eff5d499b75f09a7f"}
Sep 10 19:54:42.595329 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:42.595295    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cbcc9" event={"ID":"657fa165-c6ad-4e6a-8e91-867b848fead3","Type":"ContainerStarted","Data":"4dc9b51381a348ea5b6d5f38cfe675ca6f6f4b11b260e6e43e28f7d1bd48f515"}
Sep 10 19:54:42.611398 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:42.611340    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cbcc9" podStartSLOduration=5.615592165 podStartE2EDuration="23.611303442s" totalImagesPullingTime="17.995711277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:54:42.610567226 +0000 UTC m=+23.661095216" watchObservedRunningTime="2026-09-10 19:54:42.611303442 +0000 UTC m=+23.661831430"
Sep 10 19:54:42.729766 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:42.729687    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:42.831036 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:42.830998    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:43.196986 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:43.196685    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gdv2q"
Sep 10 19:54:43.197291 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:43.197268    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gdv2q"
Sep 10 19:54:43.398561 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:43.398517    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:43.398729 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:43.398569    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:43.398729 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:43.398647    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:43.398844 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:43.398791    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:43.602337 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:43.602245    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr" event={"ID":"e4091e39-1788-47af-9032-d55ab85e4089","Type":"ContainerStarted","Data":"0dcc3ca13d04db01a32de29b0adf4f0711e9c7c01540126fac83dbf67e190f43"}
Sep 10 19:54:44.398121 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:44.398076    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:44.398336 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:44.398221    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:44.606869 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:44.606828    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz" event={"ID":"92385da7-5b0b-4ab6-b9e0-a240595829d4","Type":"ContainerStarted","Data":"9586ab7a0fa5633e06a012b924a6f8b99bdc68f89b0074c52e1f3a57a9623987"}
Sep 10 19:54:44.606869 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:44.606857    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 19:54:44.624964 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:44.624913    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4qpz" podStartSLOduration=4.936599779 podStartE2EDuration="25.624897184s" totalImagesPullingTime="20.688297405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:54:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:54:44.62438154 +0000 UTC m=+25.674909531" watchObservedRunningTime="2026-09-10 19:54:44.624897184 +0000 UTC m=+25.675425173"
Sep 10 19:54:45.398649 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:45.398610    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:45.398649 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:45.398650    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:45.398870 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:45.398755    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:45.398940 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:45.398866    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:46.397770 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:46.397605    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:46.398165 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:46.397851    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:46.614918 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:46.614825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr" event={"ID":"e4091e39-1788-47af-9032-d55ab85e4089","Type":"ContainerStarted","Data":"840c9bb737bb7ce5ea3bc4903c84fe9ff86a798c25a7da04d0f205158b0d73c9"}
Sep 10 19:54:46.615060 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:46.615040    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:46.615133 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:46.615098    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:46.616724 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:46.616696    2561 generic.go:356] "Generic (PLEG): container finished" podID="16811901-cbb8-487a-ba53-94a0227ba869" containerID="ce1e6c73f6c89311f997f7520da55c8dcca5da07b34f862c1538c2841d480b22" exitCode=0
Sep 10 19:54:46.616813 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:46.616752    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6n4sz" event={"ID":"16811901-cbb8-487a-ba53-94a0227ba869","Type":"ContainerDied","Data":"ce1e6c73f6c89311f997f7520da55c8dcca5da07b34f862c1538c2841d480b22"}
Sep 10 19:54:46.631682 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:46.631655    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:46.643118 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:46.643067    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr" podStartSLOduration=9.593982609 podStartE2EDuration="27.643052829s" totalImagesPullingTime="18.04907022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:54:46.640261469 +0000 UTC m=+27.690789457" watchObservedRunningTime="2026-09-10 19:54:46.643052829 +0000 UTC m=+27.693580818"
Sep 10 19:54:47.398178 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.398141    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:47.398178 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.398183    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:47.398838 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:47.398281    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:47.398838 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:47.398434    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:47.621640 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.621600    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6n4sz" event={"ID":"16811901-cbb8-487a-ba53-94a0227ba869","Type":"ContainerStarted","Data":"a70663a326cddbb861e66b1571dcedead4fc66bf569a705cee9e3cdb60b738fd"}
Sep 10 19:54:47.622201 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.622181    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:47.637309 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.637277    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:54:47.871761 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.871723    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gdv2q"
Sep 10 19:54:47.871919 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.871855    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 19:54:47.872615 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.872598    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gdv2q"
Sep 10 19:54:47.942550 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.942502    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:47.947714 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.947689    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cfdm7"]
Sep 10 19:54:47.947851 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.947825    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:47.947973 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:47.947948    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:47.954636 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.954604    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tvxpw"]
Sep 10 19:54:47.954802 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.954787    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:47.954968 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:47.954944    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:47.955440 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.955387    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lx84z"]
Sep 10 19:54:47.955567 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:47.955521    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:47.956073 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:47.956050    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:48.626424 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:48.626394    2561 generic.go:356] "Generic (PLEG): container finished" podID="16811901-cbb8-487a-ba53-94a0227ba869" containerID="a70663a326cddbb861e66b1571dcedead4fc66bf569a705cee9e3cdb60b738fd" exitCode=0
Sep 10 19:54:48.627023 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:48.626464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6n4sz" event={"ID":"16811901-cbb8-487a-ba53-94a0227ba869","Type":"ContainerDied","Data":"a70663a326cddbb861e66b1571dcedead4fc66bf569a705cee9e3cdb60b738fd"}
Sep 10 19:54:49.398811 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:49.398600    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:49.398907 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:49.398671    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:49.398959 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:49.398906    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:49.399011 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:49.398687    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:49.399055 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:49.399017    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:49.399098 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:49.399082    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:49.631457 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:49.631426    2561 generic.go:356] "Generic (PLEG): container finished" podID="16811901-cbb8-487a-ba53-94a0227ba869" containerID="0294ec19960d4b81dfdc91aaf45b01bf980918737b8d4b36d1d9629afa90e6dd" exitCode=0
Sep 10 19:54:49.631925 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:49.631493    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6n4sz" event={"ID":"16811901-cbb8-487a-ba53-94a0227ba869","Type":"ContainerDied","Data":"0294ec19960d4b81dfdc91aaf45b01bf980918737b8d4b36d1d9629afa90e6dd"}
Sep 10 19:54:49.804112 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:49.804085    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:49.902261 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:49.902181    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:51.399559 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:51.398516    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:51.399559 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:51.398544    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:51.399559 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:51.398667    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvxpw" podUID="c51fb2f9-4a5c-4259-8087-67560d10732b"
Sep 10 19:54:51.399559 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:51.398718    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:51.399559 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:51.398844    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfdm7" podUID="9fce784d-e661-4e90-b91c-49dac8b1e3de"
Sep 10 19:54:51.399559 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:51.398972    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx84z" podUID="a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42"
Sep 10 19:54:52.631309 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:52.631277    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:52.730931 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:52.730889    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:53.091824 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.091748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:53.091980 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:53.091916    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:54:53.092035 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:53.091992    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret podName:c51fb2f9-4a5c-4259-8087-67560d10732b nodeName:}" failed. No retries permitted until 2026-09-10 19:55:25.091969503 +0000 UTC m=+66.142497475 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret") pod "global-pull-secret-syncer-tvxpw" (UID: "c51fb2f9-4a5c-4259-8087-67560d10732b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:54:53.192327 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.192293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs\") pod \"network-metrics-daemon-cfdm7\" (UID: \"9fce784d-e661-4e90-b91c-49dac8b1e3de\") " pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:53.192491 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:53.192457    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:54:53.192562 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:53.192540    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs podName:9fce784d-e661-4e90-b91c-49dac8b1e3de nodeName:}" failed. No retries permitted until 2026-09-10 19:55:25.192509991 +0000 UTC m=+66.243037960 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs") pod "network-metrics-daemon-cfdm7" (UID: "9fce784d-e661-4e90-b91c-49dac8b1e3de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:54:53.292955 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.292911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rfn\" (UniqueName: \"kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn\") pod \"network-check-target-lx84z\" (UID: \"a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42\") " pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:53.293146 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:53.293046    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:54:53.293146 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:53.293071    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:54:53.293146 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:53.293085    2561 projected.go:196] Error preparing data for projected volume kube-api-access-t5rfn for pod openshift-network-diagnostics/network-check-target-lx84z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:54:53.293272 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:53.293159    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn podName:a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:25.293138199 +0000 UTC m=+66.343666190 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t5rfn" (UniqueName: "kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn") pod "network-check-target-lx84z" (UID: "a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:54:53.305798 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.305774    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeReady"
Sep 10 19:54:53.305950 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.305912    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 19:54:53.380194 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.380108    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6lsgj"]
Sep 10 19:54:53.408434 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.408402    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:53.410936 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.410913    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 19:54:53.411067 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.410920    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 19:54:53.411300 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.411283    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ngfwp\""
Sep 10 19:54:53.412077 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.412058    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t4hc6"]
Sep 10 19:54:53.412959 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.412929    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:54:53.413485 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.413467    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:54:53.414515 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.414495    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:54:53.416617 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.415680    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 19:54:53.416617 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.416331    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 19:54:53.416617 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.416406    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mcqxm\""
Sep 10 19:54:53.416755 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.416620    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 19:54:53.416802 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.416765    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 19:54:53.417223 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.417189    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mr4n2\""
Sep 10 19:54:53.424897 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.424873    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t4hc6"]
Sep 10 19:54:53.425014 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.424910    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6lsgj"]
Sep 10 19:54:53.425014 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.425005    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t4hc6"
Sep 10 19:54:53.427465 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.427449    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bxvfx\""
Sep 10 19:54:53.427632 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.427609    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 19:54:53.427740 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.427728    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 19:54:53.427871 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.427855    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 19:54:53.493819 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.493777    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bshrs\" (UniqueName: \"kubernetes.io/projected/4ae9e838-42d8-479a-a2c3-fd07710f2a34-kube-api-access-bshrs\") pod \"ingress-canary-t4hc6\" (UID: \"4ae9e838-42d8-479a-a2c3-fd07710f2a34\") " pod="openshift-ingress-canary/ingress-canary-t4hc6"
Sep 10 19:54:53.494034 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.493826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfggq\" (UniqueName: \"kubernetes.io/projected/34f0063f-49f4-483d-9d24-acd1be984aee-kube-api-access-gfggq\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:53.494034 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.493867    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34f0063f-49f4-483d-9d24-acd1be984aee-config-volume\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:53.494034 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.493884    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:53.494034 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.493904    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f0063f-49f4-483d-9d24-acd1be984aee-tmp-dir\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:53.494034 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.493939    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ae9e838-42d8-479a-a2c3-fd07710f2a34-cert\") pod \"ingress-canary-t4hc6\" (UID: \"4ae9e838-42d8-479a-a2c3-fd07710f2a34\") " pod="openshift-ingress-canary/ingress-canary-t4hc6"
Sep 10 19:54:53.537011 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.536975    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:53.594955 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.594924    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bshrs\" (UniqueName: \"kubernetes.io/projected/4ae9e838-42d8-479a-a2c3-fd07710f2a34-kube-api-access-bshrs\") pod \"ingress-canary-t4hc6\" (UID: \"4ae9e838-42d8-479a-a2c3-fd07710f2a34\") " pod="openshift-ingress-canary/ingress-canary-t4hc6"
Sep 10 19:54:53.594955 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.594961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfggq\" (UniqueName: \"kubernetes.io/projected/34f0063f-49f4-483d-9d24-acd1be984aee-kube-api-access-gfggq\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:53.595167 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.595004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34f0063f-49f4-483d-9d24-acd1be984aee-config-volume\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:53.595167 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.595020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:53.595167 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.595041    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f0063f-49f4-483d-9d24-acd1be984aee-tmp-dir\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:53.595167 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.595099    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ae9e838-42d8-479a-a2c3-fd07710f2a34-cert\") pod \"ingress-canary-t4hc6\" (UID: \"4ae9e838-42d8-479a-a2c3-fd07710f2a34\") " pod="openshift-ingress-canary/ingress-canary-t4hc6"
Sep 10 19:54:53.595347 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:53.595183    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 19:54:53.595347 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:53.595265    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls podName:34f0063f-49f4-483d-9d24-acd1be984aee nodeName:}" failed. No retries permitted until 2026-09-10 19:54:54.095245191 +0000 UTC m=+35.145773173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls") pod "dns-default-6lsgj" (UID: "34f0063f-49f4-483d-9d24-acd1be984aee") : secret "dns-default-metrics-tls" not found
Sep 10 19:54:53.595501 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.595479    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f0063f-49f4-483d-9d24-acd1be984aee-tmp-dir\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:53.595776 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.595749    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34f0063f-49f4-483d-9d24-acd1be984aee-config-volume\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:53.599577 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.599555    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ae9e838-42d8-479a-a2c3-fd07710f2a34-cert\") pod \"ingress-canary-t4hc6\" (UID: \"4ae9e838-42d8-479a-a2c3-fd07710f2a34\") " pod="openshift-ingress-canary/ingress-canary-t4hc6"
Sep 10 19:54:53.600392 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.600369    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:53.604340 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.604323    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfggq\" (UniqueName: \"kubernetes.io/projected/34f0063f-49f4-483d-9d24-acd1be984aee-kube-api-access-gfggq\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:53.604753 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.604734    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bshrs\" (UniqueName: \"kubernetes.io/projected/4ae9e838-42d8-479a-a2c3-fd07710f2a34-kube-api-access-bshrs\") pod \"ingress-canary-t4hc6\" (UID: \"4ae9e838-42d8-479a-a2c3-fd07710f2a34\") " pod="openshift-ingress-canary/ingress-canary-t4hc6"
Sep 10 19:54:53.751297 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:53.751224    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t4hc6"
Sep 10 19:54:54.100475 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:54.100289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:54.100730 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:54.100475    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 19:54:54.100730 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:54.100576    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls podName:34f0063f-49f4-483d-9d24-acd1be984aee nodeName:}" failed. No retries permitted until 2026-09-10 19:54:55.100553388 +0000 UTC m=+36.151081360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls") pod "dns-default-6lsgj" (UID: "34f0063f-49f4-483d-9d24-acd1be984aee") : secret "dns-default-metrics-tls" not found
Sep 10 19:54:54.163219 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:54.163188    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t4hc6"]
Sep 10 19:54:54.183433 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:54:54.183405    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae9e838_42d8_479a_a2c3_fd07710f2a34.slice/crio-a6384486e7260c991fdf350dbc58889d492e4a99d9abd7e172652752fd3424ef WatchSource:0}: Error finding container a6384486e7260c991fdf350dbc58889d492e4a99d9abd7e172652752fd3424ef: Status 404 returned error can't find the container with id a6384486e7260c991fdf350dbc58889d492e4a99d9abd7e172652752fd3424ef
Sep 10 19:54:54.279463 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:54.279385    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:54.380671 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:54.380639    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:54.646073 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:54.645993    2561 generic.go:356] "Generic (PLEG): container finished" podID="16811901-cbb8-487a-ba53-94a0227ba869" containerID="101e46d09a7221adddbadf91769a83ff097bb3a3d8ba19a926d318537b3a4dff" exitCode=0
Sep 10 19:54:54.646242 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:54.646084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6n4sz" event={"ID":"16811901-cbb8-487a-ba53-94a0227ba869","Type":"ContainerDied","Data":"101e46d09a7221adddbadf91769a83ff097bb3a3d8ba19a926d318537b3a4dff"}
Sep 10 19:54:54.647128 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:54.647111    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t4hc6" event={"ID":"4ae9e838-42d8-479a-a2c3-fd07710f2a34","Type":"ContainerStarted","Data":"a6384486e7260c991fdf350dbc58889d492e4a99d9abd7e172652752fd3424ef"}
Sep 10 19:54:54.878668 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:54.878637    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:54.938426 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:54.938354    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:55.110019 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:55.109977    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:55.110188 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:55.110139    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 19:54:55.110272 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:55.110225    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls podName:34f0063f-49f4-483d-9d24-acd1be984aee nodeName:}" failed. No retries permitted until 2026-09-10 19:54:57.110201559 +0000 UTC m=+38.160729546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls") pod "dns-default-6lsgj" (UID: "34f0063f-49f4-483d-9d24-acd1be984aee") : secret "dns-default-metrics-tls" not found
Sep 10 19:54:55.656311 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:55.656276    2561 generic.go:356] "Generic (PLEG): container finished" podID="16811901-cbb8-487a-ba53-94a0227ba869" containerID="2857e5c3146655e56fd02d15ca61efb205b8ae0bc5e48c97e5fdbde49d7032bd" exitCode=0
Sep 10 19:54:55.656512 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:55.656361    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6n4sz" event={"ID":"16811901-cbb8-487a-ba53-94a0227ba869","Type":"ContainerDied","Data":"2857e5c3146655e56fd02d15ca61efb205b8ae0bc5e48c97e5fdbde49d7032bd"}
Sep 10 19:54:56.664137 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:56.664112    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6n4sz" event={"ID":"16811901-cbb8-487a-ba53-94a0227ba869","Type":"ContainerStarted","Data":"50a5396125d81e75bfae94d190eab1dd74a2cfa13cb5d32ee18c6dbb0dfe3d34"}
Sep 10 19:54:56.686523 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:56.686470    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6n4sz" podStartSLOduration=9.444591715 podStartE2EDuration="37.686454427s" totalImagesPullingTime="28.241862712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:54:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:54:56.684992052 +0000 UTC m=+37.735520041" watchObservedRunningTime="2026-09-10 19:54:56.686454427 +0000 UTC m=+37.736982416"
Sep 10 19:54:56.807088 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:56.806839    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:56.866711 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:56.866682    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:54:57.125979 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:57.125942    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:54:57.126157 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:57.126067    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 19:54:57.126157 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:54:57.126119    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls podName:34f0063f-49f4-483d-9d24-acd1be984aee nodeName:}" failed. No retries permitted until 2026-09-10 19:55:01.126106179 +0000 UTC m=+42.176634159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls") pod "dns-default-6lsgj" (UID: "34f0063f-49f4-483d-9d24-acd1be984aee") : secret "dns-default-metrics-tls" not found
Sep 10 19:54:57.668381 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:57.668296    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t4hc6" event={"ID":"4ae9e838-42d8-479a-a2c3-fd07710f2a34","Type":"ContainerStarted","Data":"70f6ce931f1d1e130a7d6132f96b9475f20c300d1f4e4be16add228e23091926"}
Sep 10 19:54:57.683758 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:57.683715    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t4hc6" podStartSLOduration=2.226908552 podStartE2EDuration="4.683699428s" totalImagesPullingTime="2.456790876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:54:57.683103345 +0000 UTC m=+38.733631357" watchObservedRunningTime="2026-09-10 19:54:57.683699428 +0000 UTC m=+38.734227417"
Sep 10 19:54:58.802865 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:54:58.802826    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:01.152691 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:01.152651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:55:01.153082 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:01.152793    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 19:55:01.153082 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:01.152848    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls podName:34f0063f-49f4-483d-9d24-acd1be984aee nodeName:}" failed. No retries permitted until 2026-09-10 19:55:09.152833795 +0000 UTC m=+50.203361761 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls") pod "dns-default-6lsgj" (UID: "34f0063f-49f4-483d-9d24-acd1be984aee") : secret "dns-default-metrics-tls" not found
Sep 10 19:55:07.152413 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.152373    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vnjtm"]
Sep 10 19:55:07.187688 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.187650    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vnjtm"]
Sep 10 19:55:07.187848 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.187765    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.190307 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.190287    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 19:55:07.190307 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.190302    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 10 19:55:07.190567 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.190411    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 10 19:55:07.190567 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.190472    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2tr6j\""
Sep 10 19:55:07.191292 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.191271    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 19:55:07.197568 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.197521    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 10 19:55:07.259293 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.259258    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"]
Sep 10 19:55:07.281310 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.281270    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-f7dn5"]
Sep 10 19:55:07.281479 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.281395    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:07.284089 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.284065    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 10 19:55:07.284607 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.284588    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hvgpz\""
Sep 10 19:55:07.286173 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.286156    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 19:55:07.286173 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.286165    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 10 19:55:07.286302 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.286169    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 19:55:07.294580 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.294556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7c8de1db-4a70-469e-8f56-fecd54dde30a-snapshots\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.294675 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.294593    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtv8k\" (UniqueName: \"kubernetes.io/projected/7c8de1db-4a70-469e-8f56-fecd54dde30a-kube-api-access-mtv8k\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.294726 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.294702    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8de1db-4a70-469e-8f56-fecd54dde30a-serving-cert\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.294771 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.294731    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c8de1db-4a70-469e-8f56-fecd54dde30a-tmp\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.294771 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.294748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8de1db-4a70-469e-8f56-fecd54dde30a-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.294859 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.294828    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8de1db-4a70-469e-8f56-fecd54dde30a-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.295387 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.295374    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"]
Sep 10 19:55:07.295423 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.295398    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-f7dn5"]
Sep 10 19:55:07.295472 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.295463    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:07.297862 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.297839    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ttr79\""
Sep 10 19:55:07.297957 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.297841    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 19:55:07.297957 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.297872    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 19:55:07.298076 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.298061    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 19:55:07.298213 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.298185    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 19:55:07.303762 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.303732    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 19:55:07.357473 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.357435    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-556759f-cdwrm"]
Sep 10 19:55:07.377308 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.377276    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-556759f-cdwrm"]
Sep 10 19:55:07.377435 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.377370    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.379808 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.379781    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 19:55:07.380106 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.380087    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2xjwm\""
Sep 10 19:55:07.380217 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.380180    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 19:55:07.380263 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.380250    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 19:55:07.397653 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.397512    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 19:55:07.399554 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.397877    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8de1db-4a70-469e-8f56-fecd54dde30a-serving-cert\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.399554 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.397933    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5f652bf6-f124-4f0f-a594-773e4bfed694-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:07.399554 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.397966    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ac6a59-db3b-40a2-95f0-42f55a9156d1-serving-cert\") pod \"console-operator-85b454874b-f7dn5\" (UID: \"d7ac6a59-db3b-40a2-95f0-42f55a9156d1\") " pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:07.399554 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.398003    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7ac6a59-db3b-40a2-95f0-42f55a9156d1-config\") pod \"console-operator-85b454874b-f7dn5\" (UID: \"d7ac6a59-db3b-40a2-95f0-42f55a9156d1\") " pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:07.399554 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.398042    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c8de1db-4a70-469e-8f56-fecd54dde30a-tmp\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.399554 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.398072    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8de1db-4a70-469e-8f56-fecd54dde30a-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.399554 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.398153    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:07.399554 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.398181    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7ac6a59-db3b-40a2-95f0-42f55a9156d1-trusted-ca\") pod \"console-operator-85b454874b-f7dn5\" (UID: \"d7ac6a59-db3b-40a2-95f0-42f55a9156d1\") " pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:07.399554 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.398216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8de1db-4a70-469e-8f56-fecd54dde30a-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.399554 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.398250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncjg7\" (UniqueName: \"kubernetes.io/projected/d7ac6a59-db3b-40a2-95f0-42f55a9156d1-kube-api-access-ncjg7\") pod \"console-operator-85b454874b-f7dn5\" (UID: \"d7ac6a59-db3b-40a2-95f0-42f55a9156d1\") " pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:07.399554 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.398285    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7c8de1db-4a70-469e-8f56-fecd54dde30a-snapshots\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.399554 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.398316    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtv8k\" (UniqueName: \"kubernetes.io/projected/7c8de1db-4a70-469e-8f56-fecd54dde30a-kube-api-access-mtv8k\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.399554 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.398376    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znbsb\" (UniqueName: \"kubernetes.io/projected/5f652bf6-f124-4f0f-a594-773e4bfed694-kube-api-access-znbsb\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:07.402630 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.399981    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c8de1db-4a70-469e-8f56-fecd54dde30a-tmp\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.402630 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.401149    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8de1db-4a70-469e-8f56-fecd54dde30a-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.402630 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.401258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8de1db-4a70-469e-8f56-fecd54dde30a-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.402630 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.401618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7c8de1db-4a70-469e-8f56-fecd54dde30a-snapshots\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.403417 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.403393    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8de1db-4a70-469e-8f56-fecd54dde30a-serving-cert\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.410337 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.410313    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtv8k\" (UniqueName: \"kubernetes.io/projected/7c8de1db-4a70-469e-8f56-fecd54dde30a-kube-api-access-mtv8k\") pod \"insights-operator-76bcd6ccf5-vnjtm\" (UID: \"7c8de1db-4a70-469e-8f56-fecd54dde30a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.420015 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.419988    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:07.490646 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.490613    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:07.497486 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.497471    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm"
Sep 10 19:55:07.498702 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.498683    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5f652bf6-f124-4f0f-a594-773e4bfed694-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:07.498753 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.498712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ac6a59-db3b-40a2-95f0-42f55a9156d1-serving-cert\") pod \"console-operator-85b454874b-f7dn5\" (UID: \"d7ac6a59-db3b-40a2-95f0-42f55a9156d1\") " pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:07.498753 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.498734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7ac6a59-db3b-40a2-95f0-42f55a9156d1-config\") pod \"console-operator-85b454874b-f7dn5\" (UID: \"d7ac6a59-db3b-40a2-95f0-42f55a9156d1\") " pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:07.498824 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.498777    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/078c47ec-76a8-41ba-9e75-d6ef705c38ec-image-registry-private-configuration\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.498915 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.498891    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:07.498951 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.498934    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7ac6a59-db3b-40a2-95f0-42f55a9156d1-trusted-ca\") pod \"console-operator-85b454874b-f7dn5\" (UID: \"d7ac6a59-db3b-40a2-95f0-42f55a9156d1\") " pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:07.498995 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.498965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/078c47ec-76a8-41ba-9e75-d6ef705c38ec-ca-trust-extracted\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.499039 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.499001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncjg7\" (UniqueName: \"kubernetes.io/projected/d7ac6a59-db3b-40a2-95f0-42f55a9156d1-kube-api-access-ncjg7\") pod \"console-operator-85b454874b-f7dn5\" (UID: \"d7ac6a59-db3b-40a2-95f0-42f55a9156d1\") " pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:07.499039 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:07.499027    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 19:55:07.499039 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.499033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-bound-sa-token\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.499190 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.499057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-certificates\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.499190 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.499081    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/078c47ec-76a8-41ba-9e75-d6ef705c38ec-trusted-ca\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.499190 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.499161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/078c47ec-76a8-41ba-9e75-d6ef705c38ec-installation-pull-secrets\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.499343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.499203    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.499343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.499242    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znbsb\" (UniqueName: \"kubernetes.io/projected/5f652bf6-f124-4f0f-a594-773e4bfed694-kube-api-access-znbsb\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:07.499343 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.499302    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hsqd\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-kube-api-access-7hsqd\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.499494 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:07.499428    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls podName:5f652bf6-f124-4f0f-a594-773e4bfed694 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:07.999389712 +0000 UTC m=+49.049917693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lpbmp" (UID: "5f652bf6-f124-4f0f-a594-773e4bfed694") : secret "cluster-monitoring-operator-tls" not found
Sep 10 19:55:07.499494 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.499485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5f652bf6-f124-4f0f-a594-773e4bfed694-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:07.499616 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.499516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7ac6a59-db3b-40a2-95f0-42f55a9156d1-config\") pod \"console-operator-85b454874b-f7dn5\" (UID: \"d7ac6a59-db3b-40a2-95f0-42f55a9156d1\") " pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:07.500173 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.500152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7ac6a59-db3b-40a2-95f0-42f55a9156d1-trusted-ca\") pod \"console-operator-85b454874b-f7dn5\" (UID: \"d7ac6a59-db3b-40a2-95f0-42f55a9156d1\") " pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:07.501444 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.501419    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ac6a59-db3b-40a2-95f0-42f55a9156d1-serving-cert\") pod \"console-operator-85b454874b-f7dn5\" (UID: \"d7ac6a59-db3b-40a2-95f0-42f55a9156d1\") " pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:07.507877 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.507849    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znbsb\" (UniqueName: \"kubernetes.io/projected/5f652bf6-f124-4f0f-a594-773e4bfed694-kube-api-access-znbsb\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:07.508036 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.508015    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncjg7\" (UniqueName: \"kubernetes.io/projected/d7ac6a59-db3b-40a2-95f0-42f55a9156d1-kube-api-access-ncjg7\") pod \"console-operator-85b454874b-f7dn5\" (UID: \"d7ac6a59-db3b-40a2-95f0-42f55a9156d1\") " pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:07.600192 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.600160    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/078c47ec-76a8-41ba-9e75-d6ef705c38ec-installation-pull-secrets\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.600192 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.600199    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.600436 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.600253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hsqd\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-kube-api-access-7hsqd\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.600436 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.600283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/078c47ec-76a8-41ba-9e75-d6ef705c38ec-image-registry-private-configuration\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.600436 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.600340    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/078c47ec-76a8-41ba-9e75-d6ef705c38ec-ca-trust-extracted\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.600436 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.600362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-bound-sa-token\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.600436 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.600380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-certificates\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.600702 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:07.600433    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 19:55:07.600702 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:07.600455    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-556759f-cdwrm: secret "image-registry-tls" not found
Sep 10 19:55:07.600702 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:07.600510    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls podName:078c47ec-76a8-41ba-9e75-d6ef705c38ec nodeName:}" failed. No retries permitted until 2026-09-10 19:55:08.100489788 +0000 UTC m=+49.151017757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls") pod "image-registry-556759f-cdwrm" (UID: "078c47ec-76a8-41ba-9e75-d6ef705c38ec") : secret "image-registry-tls" not found
Sep 10 19:55:07.600702 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.600435    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/078c47ec-76a8-41ba-9e75-d6ef705c38ec-trusted-ca\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.601000 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.600976    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/078c47ec-76a8-41ba-9e75-d6ef705c38ec-ca-trust-extracted\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.601238 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.601056    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-certificates\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.601353 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.601333    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/078c47ec-76a8-41ba-9e75-d6ef705c38ec-trusted-ca\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.602795 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.602774    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/078c47ec-76a8-41ba-9e75-d6ef705c38ec-image-registry-private-configuration\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.602903 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.602846    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/078c47ec-76a8-41ba-9e75-d6ef705c38ec-installation-pull-secrets\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.603708 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.603693    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:07.609134 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.609112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hsqd\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-kube-api-access-7hsqd\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.609335 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.609319    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-bound-sa-token\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:07.792569 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.792516    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vnjtm"]
Sep 10 19:55:07.796231 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:55:07.796202    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8de1db_4a70_469e_8f56_fecd54dde30a.slice/crio-d813db874cca16ac0ad37024043059dd95c0eaf8dfbd3e491f039ef5f046c6e0 WatchSource:0}: Error finding container d813db874cca16ac0ad37024043059dd95c0eaf8dfbd3e491f039ef5f046c6e0: Status 404 returned error can't find the container with id d813db874cca16ac0ad37024043059dd95c0eaf8dfbd3e491f039ef5f046c6e0
Sep 10 19:55:07.900831 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:07.900797    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-f7dn5"]
Sep 10 19:55:07.903919 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:55:07.903896    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ac6a59_db3b_40a2_95f0_42f55a9156d1.slice/crio-a2d5f6c59c86dc98d03ac5f068592cb12d9f3a966fbb3220b48a102d4637662e WatchSource:0}: Error finding container a2d5f6c59c86dc98d03ac5f068592cb12d9f3a966fbb3220b48a102d4637662e: Status 404 returned error can't find the container with id a2d5f6c59c86dc98d03ac5f068592cb12d9f3a966fbb3220b48a102d4637662e
Sep 10 19:55:08.005021 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:08.004928    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:08.005169 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:08.005041    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 19:55:08.005169 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:08.005100    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls podName:5f652bf6-f124-4f0f-a594-773e4bfed694 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:09.005085856 +0000 UTC m=+50.055613823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lpbmp" (UID: "5f652bf6-f124-4f0f-a594-773e4bfed694") : secret "cluster-monitoring-operator-tls" not found
Sep 10 19:55:08.105383 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:08.105350    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:08.105566 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:08.105504    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 19:55:08.105566 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:08.105540    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-556759f-cdwrm: secret "image-registry-tls" not found
Sep 10 19:55:08.105647 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:08.105597    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls podName:078c47ec-76a8-41ba-9e75-d6ef705c38ec nodeName:}" failed. No retries permitted until 2026-09-10 19:55:09.105581395 +0000 UTC m=+50.156109363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls") pod "image-registry-556759f-cdwrm" (UID: "078c47ec-76a8-41ba-9e75-d6ef705c38ec") : secret "image-registry-tls" not found
Sep 10 19:55:08.695679 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:08.695626    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm" event={"ID":"7c8de1db-4a70-469e-8f56-fecd54dde30a","Type":"ContainerStarted","Data":"d813db874cca16ac0ad37024043059dd95c0eaf8dfbd3e491f039ef5f046c6e0"}
Sep 10 19:55:08.697272 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:08.697244    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-f7dn5" event={"ID":"d7ac6a59-db3b-40a2-95f0-42f55a9156d1","Type":"ContainerStarted","Data":"a2d5f6c59c86dc98d03ac5f068592cb12d9f3a966fbb3220b48a102d4637662e"}
Sep 10 19:55:09.013417 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:09.013325    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:09.013577 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:09.013519    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 19:55:09.013671 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:09.013626    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls podName:5f652bf6-f124-4f0f-a594-773e4bfed694 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:11.013604017 +0000 UTC m=+52.064132002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lpbmp" (UID: "5f652bf6-f124-4f0f-a594-773e4bfed694") : secret "cluster-monitoring-operator-tls" not found
Sep 10 19:55:09.113977 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:09.113936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:09.114147 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:09.114099    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 19:55:09.114147 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:09.114123    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-556759f-cdwrm: secret "image-registry-tls" not found
Sep 10 19:55:09.114251 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:09.114193    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls podName:078c47ec-76a8-41ba-9e75-d6ef705c38ec nodeName:}" failed. No retries permitted until 2026-09-10 19:55:11.114173456 +0000 UTC m=+52.164701438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls") pod "image-registry-556759f-cdwrm" (UID: "078c47ec-76a8-41ba-9e75-d6ef705c38ec") : secret "image-registry-tls" not found
Sep 10 19:55:09.145908 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:09.145873    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:09.214812 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:09.214779    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:55:09.214962 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:09.214927    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 19:55:09.215005 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:09.214991    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls podName:34f0063f-49f4-483d-9d24-acd1be984aee nodeName:}" failed. No retries permitted until 2026-09-10 19:55:25.214976487 +0000 UTC m=+66.265504457 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls") pod "dns-default-6lsgj" (UID: "34f0063f-49f4-483d-9d24-acd1be984aee") : secret "dns-default-metrics-tls" not found
Sep 10 19:55:09.245545 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:09.245466    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:10.810239 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:10.810212    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:10.854009 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:10.853979    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-72k6n"]
Sep 10 19:55:10.856055 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:10.856037    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-72k6n"
Sep 10 19:55:10.858810 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:10.858782    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-75jnj\""
Sep 10 19:55:10.859006 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:10.858981    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 19:55:10.859068 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:10.858981    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 19:55:10.869363 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:10.869332    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-72k6n"]
Sep 10 19:55:10.896004 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:10.894950    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:10.930221 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:10.930190    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qwgw\" (UniqueName: \"kubernetes.io/projected/bd7ca140-2403-42d3-92b8-df81800237d3-kube-api-access-2qwgw\") pod \"migrator-fdbb8fdff-72k6n\" (UID: \"bd7ca140-2403-42d3-92b8-df81800237d3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-72k6n"
Sep 10 19:55:10.971839 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:10.971759    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:11.031249 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.031211    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:11.031403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.031298    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qwgw\" (UniqueName: \"kubernetes.io/projected/bd7ca140-2403-42d3-92b8-df81800237d3-kube-api-access-2qwgw\") pod \"migrator-fdbb8fdff-72k6n\" (UID: \"bd7ca140-2403-42d3-92b8-df81800237d3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-72k6n"
Sep 10 19:55:11.031403 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:11.031360    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 19:55:11.031479 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:11.031436    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls podName:5f652bf6-f124-4f0f-a594-773e4bfed694 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:15.031419131 +0000 UTC m=+56.081947100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lpbmp" (UID: "5f652bf6-f124-4f0f-a594-773e4bfed694") : secret "cluster-monitoring-operator-tls" not found
Sep 10 19:55:11.040073 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.040041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qwgw\" (UniqueName: \"kubernetes.io/projected/bd7ca140-2403-42d3-92b8-df81800237d3-kube-api-access-2qwgw\") pod \"migrator-fdbb8fdff-72k6n\" (UID: \"bd7ca140-2403-42d3-92b8-df81800237d3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-72k6n"
Sep 10 19:55:11.132580 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.132508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:11.132751 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:11.132678    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 19:55:11.132751 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:11.132697    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-556759f-cdwrm: secret "image-registry-tls" not found
Sep 10 19:55:11.132833 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:11.132768    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls podName:078c47ec-76a8-41ba-9e75-d6ef705c38ec nodeName:}" failed. No retries permitted until 2026-09-10 19:55:15.132750241 +0000 UTC m=+56.183278232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls") pod "image-registry-556759f-cdwrm" (UID: "078c47ec-76a8-41ba-9e75-d6ef705c38ec") : secret "image-registry-tls" not found
Sep 10 19:55:11.166345 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.166292    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-72k6n"
Sep 10 19:55:11.436347 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.436314    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-72k6n"]
Sep 10 19:55:11.439337 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:55:11.439314    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7ca140_2403_42d3_92b8_df81800237d3.slice/crio-c8d7e5d1cf267d4c4d92c7ca574e83bfa6cfd49c9842e169d61e7d5e8bcdefb0 WatchSource:0}: Error finding container c8d7e5d1cf267d4c4d92c7ca574e83bfa6cfd49c9842e169d61e7d5e8bcdefb0: Status 404 returned error can't find the container with id c8d7e5d1cf267d4c4d92c7ca574e83bfa6cfd49c9842e169d61e7d5e8bcdefb0
Sep 10 19:55:11.706069 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.705982    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-72k6n" event={"ID":"bd7ca140-2403-42d3-92b8-df81800237d3","Type":"ContainerStarted","Data":"c8d7e5d1cf267d4c4d92c7ca574e83bfa6cfd49c9842e169d61e7d5e8bcdefb0"}
Sep 10 19:55:11.707774 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.707752    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/0.log"
Sep 10 19:55:11.707923 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.707786    2561 generic.go:356] "Generic (PLEG): container finished" podID="d7ac6a59-db3b-40a2-95f0-42f55a9156d1" containerID="dcae8bc58cc2698d564a8477055e134e8ae3421c63f14f4d5c49bba3c0789874" exitCode=255
Sep 10 19:55:11.707923 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.707853    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-f7dn5" event={"ID":"d7ac6a59-db3b-40a2-95f0-42f55a9156d1","Type":"ContainerDied","Data":"dcae8bc58cc2698d564a8477055e134e8ae3421c63f14f4d5c49bba3c0789874"}
Sep 10 19:55:11.708202 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.708177    2561 scope.go:117] "RemoveContainer" containerID="dcae8bc58cc2698d564a8477055e134e8ae3421c63f14f4d5c49bba3c0789874"
Sep 10 19:55:11.710626 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.710601    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm" event={"ID":"7c8de1db-4a70-469e-8f56-fecd54dde30a","Type":"ContainerStarted","Data":"d7220ff41278b079c972b2dcb950526fa7056296027eaa8c4e653aeaeeb795bf"}
Sep 10 19:55:11.752709 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.752643    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm" podStartSLOduration=1.834401401 podStartE2EDuration="4.752620071s" totalImagesPullingTime="2.91821867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:11.751574217 +0000 UTC m=+52.802102207" watchObservedRunningTime="2026-09-10 19:55:11.752620071 +0000 UTC m=+52.803148061"
Sep 10 19:55:11.812730 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.812690    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z7mxd"]
Sep 10 19:55:11.815039 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.815014    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:11.817950 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.817923    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 19:55:11.818080 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.817956    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qkwps\""
Sep 10 19:55:11.818127 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.818087    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 19:55:11.830218 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.830188    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:11.833379 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.833351    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z7mxd"]
Sep 10 19:55:11.838068 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.838042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:11.838153 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.838088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4433866a-197a-4fdf-abc5-6c575d5eea05-data-volume\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:11.838207 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.838166    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4d8v\" (UniqueName: \"kubernetes.io/projected/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-api-access-n4d8v\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:11.838207 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.838187    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:11.838288 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.838220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4433866a-197a-4fdf-abc5-6c575d5eea05-crio-socket\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:11.939185 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.939121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:11.939185 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.939180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4433866a-197a-4fdf-abc5-6c575d5eea05-data-volume\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:11.939406 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.939243    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4d8v\" (UniqueName: \"kubernetes.io/projected/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-api-access-n4d8v\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:11.939406 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.939261    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:11.939406 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.939299    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4433866a-197a-4fdf-abc5-6c575d5eea05-crio-socket\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:11.939406 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:11.939310    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 19:55:11.939406 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.939378    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4433866a-197a-4fdf-abc5-6c575d5eea05-crio-socket\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:11.939406 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:11.939377    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 19:55:11.939621 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:11.939396    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls podName:4433866a-197a-4fdf-abc5-6c575d5eea05 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:12.439377528 +0000 UTC m=+53.489905506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7mxd" (UID: "4433866a-197a-4fdf-abc5-6c575d5eea05") : secret "insights-runtime-extractor-tls" not found
Sep 10 19:55:11.939621 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:11.939468    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm podName:4433866a-197a-4fdf-abc5-6c575d5eea05 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:12.439452281 +0000 UTC m=+53.489980248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7mxd" (UID: "4433866a-197a-4fdf-abc5-6c575d5eea05") : configmap "kube-rbac-proxy" not found
Sep 10 19:55:11.939709 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.939619    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4433866a-197a-4fdf-abc5-6c575d5eea05-data-volume\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:11.948375 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.948349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4d8v\" (UniqueName: \"kubernetes.io/projected/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-api-access-n4d8v\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:11.957171 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:11.957111    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:12.203545 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.203490    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:12.268103 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.268073    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:12.444169 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.444122    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:12.444413 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:12.444257    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 19:55:12.444413 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.444268    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:12.444413 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:12.444334    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm podName:4433866a-197a-4fdf-abc5-6c575d5eea05 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:13.444312304 +0000 UTC m=+54.494840302 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7mxd" (UID: "4433866a-197a-4fdf-abc5-6c575d5eea05") : configmap "kube-rbac-proxy" not found
Sep 10 19:55:12.444413 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:12.444411    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 19:55:12.444669 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:12.444465    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls podName:4433866a-197a-4fdf-abc5-6c575d5eea05 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:13.444449773 +0000 UTC m=+54.494977758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7mxd" (UID: "4433866a-197a-4fdf-abc5-6c575d5eea05") : secret "insights-runtime-extractor-tls" not found
Sep 10 19:55:12.715371 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.715344    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 19:55:12.715804 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.715789    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/0.log"
Sep 10 19:55:12.715894 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.715826    2561 generic.go:356] "Generic (PLEG): container finished" podID="d7ac6a59-db3b-40a2-95f0-42f55a9156d1" containerID="c72e656630bedd2d1374fb95035f464a6f70aab9e748c76d216fe26b1f98f23e" exitCode=255
Sep 10 19:55:12.715944 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.715908    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-f7dn5" event={"ID":"d7ac6a59-db3b-40a2-95f0-42f55a9156d1","Type":"ContainerDied","Data":"c72e656630bedd2d1374fb95035f464a6f70aab9e748c76d216fe26b1f98f23e"}
Sep 10 19:55:12.716001 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.715989    2561 scope.go:117] "RemoveContainer" containerID="dcae8bc58cc2698d564a8477055e134e8ae3421c63f14f4d5c49bba3c0789874"
Sep 10 19:55:12.716215 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.716191    2561 scope.go:117] "RemoveContainer" containerID="c72e656630bedd2d1374fb95035f464a6f70aab9e748c76d216fe26b1f98f23e"
Sep 10 19:55:12.716406 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:12.716381    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-f7dn5_openshift-console-operator(d7ac6a59-db3b-40a2-95f0-42f55a9156d1)\"" pod="openshift-console-operator/console-operator-85b454874b-f7dn5" podUID="d7ac6a59-db3b-40a2-95f0-42f55a9156d1"
Sep 10 19:55:12.717862 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.717838    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-72k6n" event={"ID":"bd7ca140-2403-42d3-92b8-df81800237d3","Type":"ContainerStarted","Data":"be59113e27b3af7ee6c4ba301997be259c20b032cb7b0c13d653f047fc10452a"}
Sep 10 19:55:12.717992 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.717888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-72k6n" event={"ID":"bd7ca140-2403-42d3-92b8-df81800237d3","Type":"ContainerStarted","Data":"574c8c93f7d774f957a9d35617b2fc04f88761ee689dae70d2f1bca244b239ea"}
Sep 10 19:55:12.718401 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.718376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:12.755077 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.755014    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-72k6n" podStartSLOduration=1.64462947 podStartE2EDuration="2.754997296s" totalImagesPullingTime="1.110367826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:12.75430058 +0000 UTC m=+53.804828569" watchObservedRunningTime="2026-09-10 19:55:12.754997296 +0000 UTC m=+53.805525286"
Sep 10 19:55:12.818793 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.818709    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:12.926187 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:12.926152    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:13.023884 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:13.023851    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:13.453018 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:13.452977    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:13.453203 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:13.453074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:13.453203 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:13.453148    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 19:55:13.453203 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:13.453196    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 19:55:13.453308 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:13.453208    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls podName:4433866a-197a-4fdf-abc5-6c575d5eea05 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:15.45319393 +0000 UTC m=+56.503721900 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7mxd" (UID: "4433866a-197a-4fdf-abc5-6c575d5eea05") : secret "insights-runtime-extractor-tls" not found
Sep 10 19:55:13.453308 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:13.453248    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm podName:4433866a-197a-4fdf-abc5-6c575d5eea05 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:15.453235041 +0000 UTC m=+56.503763050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7mxd" (UID: "4433866a-197a-4fdf-abc5-6c575d5eea05") : configmap "kube-rbac-proxy" not found
Sep 10 19:55:13.722163 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:13.722085    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 19:55:13.722423 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:13.722407    2561 scope.go:117] "RemoveContainer" containerID="c72e656630bedd2d1374fb95035f464a6f70aab9e748c76d216fe26b1f98f23e"
Sep 10 19:55:13.722601 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:13.722581    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-f7dn5_openshift-console-operator(d7ac6a59-db3b-40a2-95f0-42f55a9156d1)\"" pod="openshift-console-operator/console-operator-85b454874b-f7dn5" podUID="d7ac6a59-db3b-40a2-95f0-42f55a9156d1"
Sep 10 19:55:14.537183 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:14.537148    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:14.562896 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:14.562867    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tb2sv_0e80f59a-083f-401d-92ba-ae440fb4840e/dns-node-resolver/0.log"
Sep 10 19:55:14.597398 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:14.597366    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:15.067037 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:15.066998    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:15.067219 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:15.067126    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 19:55:15.067219 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:15.067189    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls podName:5f652bf6-f124-4f0f-a594-773e4bfed694 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:23.067174836 +0000 UTC m=+64.117702807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lpbmp" (UID: "5f652bf6-f124-4f0f-a594-773e4bfed694") : secret "cluster-monitoring-operator-tls" not found
Sep 10 19:55:15.168291 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:15.168253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:15.168435 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:15.168402    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 19:55:15.168435 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:15.168422    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-556759f-cdwrm: secret "image-registry-tls" not found
Sep 10 19:55:15.168500 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:15.168478    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls podName:078c47ec-76a8-41ba-9e75-d6ef705c38ec nodeName:}" failed. No retries permitted until 2026-09-10 19:55:23.168462678 +0000 UTC m=+64.218990648 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls") pod "image-registry-556759f-cdwrm" (UID: "078c47ec-76a8-41ba-9e75-d6ef705c38ec") : secret "image-registry-tls" not found
Sep 10 19:55:15.470817 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:15.470708    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:15.470817 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:15.470799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:15.471011 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:15.470860    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 19:55:15.471011 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:15.470885    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 19:55:15.471011 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:15.470931    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls podName:4433866a-197a-4fdf-abc5-6c575d5eea05 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:19.470915228 +0000 UTC m=+60.521443198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7mxd" (UID: "4433866a-197a-4fdf-abc5-6c575d5eea05") : secret "insights-runtime-extractor-tls" not found
Sep 10 19:55:15.471011 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:15.470945    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm podName:4433866a-197a-4fdf-abc5-6c575d5eea05 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:19.47093923 +0000 UTC m=+60.521467197 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7mxd" (UID: "4433866a-197a-4fdf-abc5-6c575d5eea05") : configmap "kube-rbac-proxy" not found
Sep 10 19:55:15.488887 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:15.488859    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:15.563481 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:15.563453    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tmzvv_2d6179f7-5886-457f-9b86-a9849a6f3eaa/node-ca/0.log"
Sep 10 19:55:16.563818 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:16.563791    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t4hc6_4ae9e838-42d8-479a-a2c3-fd07710f2a34/serve-healthcheck-canary/0.log"
Sep 10 19:55:16.764280 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:16.764249    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-72k6n_bd7ca140-2403-42d3-92b8-df81800237d3/migrator/0.log"
Sep 10 19:55:16.963073 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:16.963005    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-72k6n_bd7ca140-2403-42d3-92b8-df81800237d3/graceful-termination/0.log"
Sep 10 19:55:17.604710 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:17.604672    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:17.605088 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:17.604728    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:17.605088 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:17.605061    2561 scope.go:117] "RemoveContainer" containerID="c72e656630bedd2d1374fb95035f464a6f70aab9e748c76d216fe26b1f98f23e"
Sep 10 19:55:17.605231 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:17.605211    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-f7dn5_openshift-console-operator(d7ac6a59-db3b-40a2-95f0-42f55a9156d1)\"" pod="openshift-console-operator/console-operator-85b454874b-f7dn5" podUID="d7ac6a59-db3b-40a2-95f0-42f55a9156d1"
Sep 10 19:55:18.061453 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:18.061412    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:18.202295 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:18.202243    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:18.264403 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:18.264376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:18.348941 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:18.348822    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:18.472165 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:18.472135    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:18.756700 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:18.756606    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:19.167616 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:19.167581    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:19.504768 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:19.504685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:19.504914 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:19.504773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:19.504914 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:19.504801    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 19:55:19.504914 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:19.504862    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm podName:4433866a-197a-4fdf-abc5-6c575d5eea05 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:27.504845669 +0000 UTC m=+68.555373635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7mxd" (UID: "4433866a-197a-4fdf-abc5-6c575d5eea05") : configmap "kube-rbac-proxy" not found
Sep 10 19:55:19.504914 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:19.504872    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 19:55:19.505058 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:19.504973    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls podName:4433866a-197a-4fdf-abc5-6c575d5eea05 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:27.504959933 +0000 UTC m=+68.555487900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7mxd" (UID: "4433866a-197a-4fdf-abc5-6c575d5eea05") : secret "insights-runtime-extractor-tls" not found
Sep 10 19:55:19.643344 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:19.643310    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xstzr"
Sep 10 19:55:19.984286 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:19.984254    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:20.176897 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:20.176865    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:20.357101 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:20.357066    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:20.567113 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:20.567077    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:20.761615 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:20.761514    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:21.544979 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:21.544948    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:21.747950 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:21.747915    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:21.808960 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:21.808874    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:21.959841 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:21.959787    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:22.353354 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:22.353327    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:22.547852 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:22.547805    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:22.753710 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:22.753630    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:23.134573 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:23.134510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:23.134791 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:23.134679    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 19:55:23.134791 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:23.134760    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls podName:5f652bf6-f124-4f0f-a594-773e4bfed694 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:39.134739825 +0000 UTC m=+80.185267807 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lpbmp" (UID: "5f652bf6-f124-4f0f-a594-773e4bfed694") : secret "cluster-monitoring-operator-tls" not found
Sep 10 19:55:23.235320 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:23.235276    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:23.237835 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:23.237809    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls\") pod \"image-registry-556759f-cdwrm\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") " pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:23.288456 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:23.288423    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2xjwm\""
Sep 10 19:55:23.296391 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:23.296373    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:23.567767 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:23.567733    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-556759f-cdwrm"]
Sep 10 19:55:23.571414 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:55:23.571387    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078c47ec_76a8_41ba_9e75_d6ef705c38ec.slice/crio-522edba3fa2a45d07ab0861bec968672e709f68405792ef46d83a880a3341a52 WatchSource:0}: Error finding container 522edba3fa2a45d07ab0861bec968672e709f68405792ef46d83a880a3341a52: Status 404 returned error can't find the container with id 522edba3fa2a45d07ab0861bec968672e709f68405792ef46d83a880a3341a52
Sep 10 19:55:23.746172 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:23.746139    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:23.756953 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:23.756922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-556759f-cdwrm" event={"ID":"078c47ec-76a8-41ba-9e75-d6ef705c38ec","Type":"ContainerStarted","Data":"bdf118f8ac69dbecc50d9d3c0a15f1777b1a48f0934bdafffe969a44d6707cbf"}
Sep 10 19:55:23.757106 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:23.756967    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-556759f-cdwrm" event={"ID":"078c47ec-76a8-41ba-9e75-d6ef705c38ec","Type":"ContainerStarted","Data":"522edba3fa2a45d07ab0861bec968672e709f68405792ef46d83a880a3341a52"}
Sep 10 19:55:23.757106 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:23.757018    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:23.776590 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:23.776019    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-556759f-cdwrm" podStartSLOduration=16.776001111 podStartE2EDuration="16.776001111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:23.775061203 +0000 UTC m=+64.825589193" watchObservedRunningTime="2026-09-10 19:55:23.776001111 +0000 UTC m=+64.826529099"
Sep 10 19:55:23.947757 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:23.947670    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:24.049045 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:24.049012    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:24.157793 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:24.157761    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:24.253706 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:24.253624    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:24.358661 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:24.358621    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:25.154221 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.154183    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:55:25.157344 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.157325    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 19:55:25.167161 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.167133    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c51fb2f9-4a5c-4259-8087-67560d10732b-original-pull-secret\") pod \"global-pull-secret-syncer-tvxpw\" (UID: \"c51fb2f9-4a5c-4259-8087-67560d10732b\") " pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:55:25.182440 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.182415    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:25.230637 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.230599    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvxpw"
Sep 10 19:55:25.255219 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.255187    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs\") pod \"network-metrics-daemon-cfdm7\" (UID: \"9fce784d-e661-4e90-b91c-49dac8b1e3de\") " pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:55:25.255362 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.255233    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:55:25.257793 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.257768    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34f0063f-49f4-483d-9d24-acd1be984aee-metrics-tls\") pod \"dns-default-6lsgj\" (UID: \"34f0063f-49f4-483d-9d24-acd1be984aee\") " pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:55:25.257911 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.257893    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 19:55:25.265711 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:25.265695    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 19:55:25.265775 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:25.265762    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs podName:9fce784d-e661-4e90-b91c-49dac8b1e3de nodeName:}" failed. No retries permitted until 2026-09-10 19:56:29.265740546 +0000 UTC m=+130.316268534 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs") pod "network-metrics-daemon-cfdm7" (UID: "9fce784d-e661-4e90-b91c-49dac8b1e3de") : secret "metrics-daemon-secret" not found
Sep 10 19:55:25.356695 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.356656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rfn\" (UniqueName: \"kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn\") pod \"network-check-target-lx84z\" (UID: \"a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42\") " pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:55:25.359182 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.359164    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 19:55:25.369682 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.369660    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 19:55:25.380264 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.380238    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5rfn\" (UniqueName: \"kubernetes.io/projected/a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42-kube-api-access-t5rfn\") pod \"network-check-target-lx84z\" (UID: \"a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42\") " pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:55:25.423273 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.423189    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:25.525800 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.525775    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ngfwp\""
Sep 10 19:55:25.533782 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.533764    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:55:25.541907 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.541880    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mcqxm\""
Sep 10 19:55:25.551329 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.551302    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:55:25.552242 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.552220    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tvxpw"]
Sep 10 19:55:25.554959 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:55:25.554917    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51fb2f9_4a5c_4259_8087_67560d10732b.slice/crio-2368dc1262ea1edbb8bdda21539a1e4aa87e38c37302e95b49e79551609091ed WatchSource:0}: Error finding container 2368dc1262ea1edbb8bdda21539a1e4aa87e38c37302e95b49e79551609091ed: Status 404 returned error can't find the container with id 2368dc1262ea1edbb8bdda21539a1e4aa87e38c37302e95b49e79551609091ed
Sep 10 19:55:25.765770 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.765689    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tvxpw" event={"ID":"c51fb2f9-4a5c-4259-8087-67560d10732b","Type":"ContainerStarted","Data":"2368dc1262ea1edbb8bdda21539a1e4aa87e38c37302e95b49e79551609091ed"}
Sep 10 19:55:25.821348 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.821323    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6lsgj"]
Sep 10 19:55:25.823778 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:55:25.823752    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f0063f_49f4_483d_9d24_acd1be984aee.slice/crio-f152ca273872062647374ccac46685bf782eea29cb080c37a2a921d096d4bf1d WatchSource:0}: Error finding container f152ca273872062647374ccac46685bf782eea29cb080c37a2a921d096d4bf1d: Status 404 returned error can't find the container with id f152ca273872062647374ccac46685bf782eea29cb080c37a2a921d096d4bf1d
Sep 10 19:55:25.834232 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.834156    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lx84z"]
Sep 10 19:55:25.836449 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:55:25.836431    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bc9a80_7ed2_4d5f_b06b_f27fd3c83d42.slice/crio-afdfdf214c848d04e07fb5f41c3bac66eb038843ad32badcfe506fd9bee09266 WatchSource:0}: Error finding container afdfdf214c848d04e07fb5f41c3bac66eb038843ad32badcfe506fd9bee09266: Status 404 returned error can't find the container with id afdfdf214c848d04e07fb5f41c3bac66eb038843ad32badcfe506fd9bee09266
Sep 10 19:55:25.963480 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:25.963445    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:26.357611 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:26.357483    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:26.760846 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:26.760670    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:26.772311 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:26.771981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lx84z" event={"ID":"a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42","Type":"ContainerStarted","Data":"afdfdf214c848d04e07fb5f41c3bac66eb038843ad32badcfe506fd9bee09266"}
Sep 10 19:55:26.773731 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:26.773701    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6lsgj" event={"ID":"34f0063f-49f4-483d-9d24-acd1be984aee","Type":"ContainerStarted","Data":"f152ca273872062647374ccac46685bf782eea29cb080c37a2a921d096d4bf1d"}
Sep 10 19:55:26.962915 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:26.962862    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:27.156932 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:27.156894    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:27.575229 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:27.575189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:27.575747 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:27.575284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:27.575747 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:27.575339    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 19:55:27.575747 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:55:27.575431    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls podName:4433866a-197a-4fdf-abc5-6c575d5eea05 nodeName:}" failed. No retries permitted until 2026-09-10 19:55:43.575408715 +0000 UTC m=+84.625936701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7mxd" (UID: "4433866a-197a-4fdf-abc5-6c575d5eea05") : secret "insights-runtime-extractor-tls" not found
Sep 10 19:55:27.575961 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:27.575934    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4433866a-197a-4fdf-abc5-6c575d5eea05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:27.807908 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:27.807871    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:28.017712 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:28.017622    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:28.358220 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:28.358178    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:28.398681 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:28.398643    2561 scope.go:117] "RemoveContainer" containerID="c72e656630bedd2d1374fb95035f464a6f70aab9e748c76d216fe26b1f98f23e"
Sep 10 19:55:28.583769 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:28.583730    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:28.767154 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:28.767122    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:28.782890 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:28.782861    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6lsgj" event={"ID":"34f0063f-49f4-483d-9d24-acd1be984aee","Type":"ContainerStarted","Data":"bf02590825644eb154bbc1bf092320cfc35b4e1ea102b0ef47729f275f09bf02"}
Sep 10 19:55:28.783082 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:28.782910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6lsgj" event={"ID":"34f0063f-49f4-483d-9d24-acd1be984aee","Type":"ContainerStarted","Data":"d03f78e809a4e88452382cdd24f2a5856e679dd0f96eae395ed0702543558d9b"}
Sep 10 19:55:28.783082 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:28.782945    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:55:28.802406 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:28.802357    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6lsgj" podStartSLOduration=33.852868478 podStartE2EDuration="35.802339602s" totalImagesPullingTime="1.949471124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:28.800615898 +0000 UTC m=+69.851143892" watchObservedRunningTime="2026-09-10 19:55:28.802339602 +0000 UTC m=+69.852867591"
Sep 10 19:55:30.797106 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:30.797068    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tvxpw" event={"ID":"c51fb2f9-4a5c-4259-8087-67560d10732b","Type":"ContainerStarted","Data":"ffa32b7da9617ad64bf73d2e56482e71b4daeb794a017ae1fae59632dde6c765"}
Sep 10 19:55:30.799105 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:30.799084    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 19:55:30.799226 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:30.799178    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-f7dn5" event={"ID":"d7ac6a59-db3b-40a2-95f0-42f55a9156d1","Type":"ContainerStarted","Data":"c26b57d67e5ba5cbd80f64664c9769de769d316f794a6a454e9ad246a965b7e8"}
Sep 10 19:55:30.799451 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:30.799431    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:30.800847 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:30.800826    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lx84z" event={"ID":"a0bc9a80-7ed2-4d5f-b06b-f27fd3c83d42","Type":"ContainerStarted","Data":"8b1d212758b9d588cdc3cda194d13ef8326491be9ac705d7bbc46b06997c757b"}
Sep 10 19:55:30.801026 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:30.801008    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:55:30.812949 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:30.812908    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tvxpw" podStartSLOduration=66.841830817 podStartE2EDuration="1m11.81289591s" totalImagesPullingTime="4.971065093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:30.812521498 +0000 UTC m=+71.863049487" watchObservedRunningTime="2026-09-10 19:55:30.81289591 +0000 UTC m=+71.863423910"
Sep 10 19:55:30.826958 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:30.826911    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lx84z" podStartSLOduration=67.135365756 podStartE2EDuration="1m11.826893515s" totalImagesPullingTime="4.691527759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:30.826872138 +0000 UTC m=+71.877400144" watchObservedRunningTime="2026-09-10 19:55:30.826893515 +0000 UTC m=+71.877421502"
Sep 10 19:55:30.844934 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:30.844885    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-f7dn5" podStartSLOduration=21.029976808 podStartE2EDuration="23.844868916s" totalImagesPullingTime="2.814892108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:30.844337249 +0000 UTC m=+71.894865239" watchObservedRunningTime="2026-09-10 19:55:30.844868916 +0000 UTC m=+71.895396954"
Sep 10 19:55:31.800072 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:31.800021    2561 patch_prober.go:28] interesting pod/console-operator-85b454874b-f7dn5 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.134.0.10:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 19:55:31.800420 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:31.800099    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-85b454874b-f7dn5" podUID="d7ac6a59-db3b-40a2-95f0-42f55a9156d1" containerName="console-operator" probeResult="failure" output="Get \"https://10.134.0.10:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 10 19:55:32.321002 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:32.320971    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-f7dn5"
Sep 10 19:55:32.499262 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:32.499224    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:32.599558 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:32.599460    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:32.657680 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:32.657647    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:32.803829 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:32.803798    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:32.989560 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:32.989466    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:33.197198 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:33.197167    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:33.266228 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:33.266161    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:33.581046 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:33.581017    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:33.641620 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:33.641592    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:34.046115 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:34.046078    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:34.123588 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:34.123556    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:34.395867 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:34.395793    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:34.586451 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:34.586420    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:34.752187 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:34.752106    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:35.612140 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:35.612108    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:37.653302 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.653273    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b96b99956-49fmc"]
Sep 10 19:55:37.656818 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.656795    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b96b99956-49fmc"
Sep 10 19:55:37.662805 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.662782    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 19:55:37.662937 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.662808    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-s46bw\""
Sep 10 19:55:37.662937 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.662782    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 19:55:37.662937 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.662918    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 19:55:37.663644 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.663625    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 19:55:37.678710 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.678686    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b96b99956-49fmc"]
Sep 10 19:55:37.758743 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.758706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c913ffe-b838-400f-a983-24294f4c9c8b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b96b99956-49fmc\" (UID: \"0c913ffe-b838-400f-a983-24294f4c9c8b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b96b99956-49fmc"
Sep 10 19:55:37.758926 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.758854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svlwn\" (UniqueName: \"kubernetes.io/projected/0c913ffe-b838-400f-a983-24294f4c9c8b-kube-api-access-svlwn\") pod \"managed-serviceaccount-addon-agent-6b96b99956-49fmc\" (UID: \"0c913ffe-b838-400f-a983-24294f4c9c8b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b96b99956-49fmc"
Sep 10 19:55:37.763643 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.763621    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-556759f-cdwrm"]
Sep 10 19:55:37.842160 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.842126    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:37.859351 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.859322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svlwn\" (UniqueName: \"kubernetes.io/projected/0c913ffe-b838-400f-a983-24294f4c9c8b-kube-api-access-svlwn\") pod \"managed-serviceaccount-addon-agent-6b96b99956-49fmc\" (UID: \"0c913ffe-b838-400f-a983-24294f4c9c8b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b96b99956-49fmc"
Sep 10 19:55:37.859351 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.859353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c913ffe-b838-400f-a983-24294f4c9c8b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b96b99956-49fmc\" (UID: \"0c913ffe-b838-400f-a983-24294f4c9c8b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b96b99956-49fmc"
Sep 10 19:55:37.862225 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.862199    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c913ffe-b838-400f-a983-24294f4c9c8b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b96b99956-49fmc\" (UID: \"0c913ffe-b838-400f-a983-24294f4c9c8b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b96b99956-49fmc"
Sep 10 19:55:37.863142 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.863123    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-577c58467f-zwvr7"]
Sep 10 19:55:37.866050 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.866031    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:37.874099 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.874077    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svlwn\" (UniqueName: \"kubernetes.io/projected/0c913ffe-b838-400f-a983-24294f4c9c8b-kube-api-access-svlwn\") pod \"managed-serviceaccount-addon-agent-6b96b99956-49fmc\" (UID: \"0c913ffe-b838-400f-a983-24294f4c9c8b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b96b99956-49fmc"
Sep 10 19:55:37.891073 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.891048    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-577c58467f-zwvr7"]
Sep 10 19:55:37.958486 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.958407    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:37.960521 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.960495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8crb\" (UniqueName: \"kubernetes.io/projected/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-kube-api-access-x8crb\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:37.960637 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.960613    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-installation-pull-secrets\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:37.960702 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.960654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-bound-sa-token\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:37.960702 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.960679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-registry-certificates\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:37.960801 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.960738    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-trusted-ca\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:37.960801 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.960768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-registry-tls\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:37.960908 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.960821    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-image-registry-private-configuration\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:37.960908 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.960872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-ca-trust-extracted\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:37.978964 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:37.978938    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b96b99956-49fmc"
Sep 10 19:55:38.061886 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.061857    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-bound-sa-token\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.062046 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.061892    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-registry-certificates\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.062046 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.061937    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-trusted-ca\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.062046 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.061957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-registry-tls\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.062046 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.061982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-image-registry-private-configuration\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.062046 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.062027    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-ca-trust-extracted\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.062255 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.062060    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8crb\" (UniqueName: \"kubernetes.io/projected/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-kube-api-access-x8crb\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.062255 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.062134    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-installation-pull-secrets\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.062472 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.062445    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-ca-trust-extracted\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.062877 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.062855    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-registry-certificates\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.063141 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.063121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-trusted-ca\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.064753 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.064734    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-registry-tls\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.065013 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.064996    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-image-registry-private-configuration\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.065157 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.065139    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-installation-pull-secrets\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.072347 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.072298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8crb\" (UniqueName: \"kubernetes.io/projected/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-kube-api-access-x8crb\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.072455 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.072384    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cadeb1ce-a05e-49ad-870b-9b4afb130bfe-bound-sa-token\") pod \"image-registry-577c58467f-zwvr7\" (UID: \"cadeb1ce-a05e-49ad-870b-9b4afb130bfe\") " pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.072500 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.072482    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:38.172158 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.172128    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:38.175128 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.175103    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.269666 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.268183    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b96b99956-49fmc"]
Sep 10 19:55:38.271389 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:55:38.271362    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c913ffe_b838_400f_a983_24294f4c9c8b.slice/crio-6ee1b8371a32613163e4bba2402c210bcf226f2d563fad4f7e6226fedb90d655 WatchSource:0}: Error finding container 6ee1b8371a32613163e4bba2402c210bcf226f2d563fad4f7e6226fedb90d655: Status 404 returned error can't find the container with id 6ee1b8371a32613163e4bba2402c210bcf226f2d563fad4f7e6226fedb90d655
Sep 10 19:55:38.451685 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.451661    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-577c58467f-zwvr7"]
Sep 10 19:55:38.453665 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:55:38.453635    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcadeb1ce_a05e_49ad_870b_9b4afb130bfe.slice/crio-38dae781fffd6407ee136ae195d10061b16f12e1c2e99541cb91af13b79593df WatchSource:0}: Error finding container 38dae781fffd6407ee136ae195d10061b16f12e1c2e99541cb91af13b79593df: Status 404 returned error can't find the container with id 38dae781fffd6407ee136ae195d10061b16f12e1c2e99541cb91af13b79593df
Sep 10 19:55:38.794770 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.794743    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6lsgj"
Sep 10 19:55:38.828602 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.828563    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b96b99956-49fmc" event={"ID":"0c913ffe-b838-400f-a983-24294f4c9c8b","Type":"ContainerStarted","Data":"6ee1b8371a32613163e4bba2402c210bcf226f2d563fad4f7e6226fedb90d655"}
Sep 10 19:55:38.830400 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.830366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-577c58467f-zwvr7" event={"ID":"cadeb1ce-a05e-49ad-870b-9b4afb130bfe","Type":"ContainerStarted","Data":"846124abc052b016cdf88b7c41f085449ed88911ce917f42c6b24e64109cd059"}
Sep 10 19:55:38.830553 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.830419    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-577c58467f-zwvr7" event={"ID":"cadeb1ce-a05e-49ad-870b-9b4afb130bfe","Type":"ContainerStarted","Data":"38dae781fffd6407ee136ae195d10061b16f12e1c2e99541cb91af13b79593df"}
Sep 10 19:55:38.830553 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.830518    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:38.856287 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:38.855944    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-577c58467f-zwvr7" podStartSLOduration=1.855921066 podStartE2EDuration="1.855921066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:38.854753629 +0000 UTC m=+79.905281621" watchObservedRunningTime="2026-09-10 19:55:38.855921066 +0000 UTC m=+79.906449055"
Sep 10 19:55:39.172599 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:39.172496    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:39.176253 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:39.176189    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f652bf6-f124-4f0f-a594-773e4bfed694-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpbmp\" (UID: \"5f652bf6-f124-4f0f-a594-773e4bfed694\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:39.395813 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:39.395778    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hvgpz\""
Sep 10 19:55:39.400332 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:39.400303    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"
Sep 10 19:55:39.690773 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:39.690731    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp"]
Sep 10 19:55:39.694308 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:55:39.694278    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f652bf6_f124_4f0f_a594_773e4bfed694.slice/crio-3311091ac5234de941b7f50d34dae59bb0e043ec87777f26ea2e016c4661c9c1 WatchSource:0}: Error finding container 3311091ac5234de941b7f50d34dae59bb0e043ec87777f26ea2e016c4661c9c1: Status 404 returned error can't find the container with id 3311091ac5234de941b7f50d34dae59bb0e043ec87777f26ea2e016c4661c9c1
Sep 10 19:55:39.835397 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:39.835349    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp" event={"ID":"5f652bf6-f124-4f0f-a594-773e4bfed694","Type":"ContainerStarted","Data":"3311091ac5234de941b7f50d34dae59bb0e043ec87777f26ea2e016c4661c9c1"}
Sep 10 19:55:41.341989 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:41.341943    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:41.429255 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:41.429206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:41.848868 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:41.848832    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b96b99956-49fmc" event={"ID":"0c913ffe-b838-400f-a983-24294f4c9c8b","Type":"ContainerStarted","Data":"007ff1b91e2ed09c13fd28a2d322d73d0d7edebbf1b02418f47e14a02de36183"}
Sep 10 19:55:41.867770 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:41.867703    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b96b99956-49fmc" podStartSLOduration=2.336807415 podStartE2EDuration="4.867686187s" totalImagesPullingTime="2.530878772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:41.866390359 +0000 UTC m=+82.916918349" watchObservedRunningTime="2026-09-10 19:55:41.867686187 +0000 UTC m=+82.918214176"
Sep 10 19:55:42.698883 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:42.698788    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:42.809154 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:42.809123    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:42.854467 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:42.854436    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp" event={"ID":"5f652bf6-f124-4f0f-a594-773e4bfed694","Type":"ContainerStarted","Data":"ced5d15b290293246412520a4ece7dec8783d2bc3d1927b361134cf3360b53f2"}
Sep 10 19:55:42.886043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:42.885986    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpbmp" podStartSLOduration=33.246063788 podStartE2EDuration="35.88596825s" totalImagesPullingTime="2.639904462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:42.885507607 +0000 UTC m=+83.936035595" watchObservedRunningTime="2026-09-10 19:55:42.88596825 +0000 UTC m=+83.936496240"
Sep 10 19:55:42.915827 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:42.915801    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:43.013726 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:43.013643    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:43.611089 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:43.611053    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:43.613550 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:43.613512    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4433866a-197a-4fdf-abc5-6c575d5eea05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7mxd\" (UID: \"4433866a-197a-4fdf-abc5-6c575d5eea05\") " pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:43.629073 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:43.629043    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qkwps\""
Sep 10 19:55:43.636955 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:43.636937    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z7mxd"
Sep 10 19:55:43.907566 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:43.907303    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z7mxd"]
Sep 10 19:55:43.910366 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:55:43.910336    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4433866a_197a_4fdf_abc5_6c575d5eea05.slice/crio-d0c25dd38d927a0d0ca2c10467ca5aaedfb25c71c50fa2f27ad973acda3ba43e WatchSource:0}: Error finding container d0c25dd38d927a0d0ca2c10467ca5aaedfb25c71c50fa2f27ad973acda3ba43e: Status 404 returned error can't find the container with id d0c25dd38d927a0d0ca2c10467ca5aaedfb25c71c50fa2f27ad973acda3ba43e
Sep 10 19:55:44.863348 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:44.863314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z7mxd" event={"ID":"4433866a-197a-4fdf-abc5-6c575d5eea05","Type":"ContainerStarted","Data":"c5dd5bf38488d417a78217cb9c2afd905893f34c0bce1fab48c119e5c6a75933"}
Sep 10 19:55:44.863511 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:44.863365    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z7mxd" event={"ID":"4433866a-197a-4fdf-abc5-6c575d5eea05","Type":"ContainerStarted","Data":"0b489fe273f105d97c18e12808cdf474b7ab5c493c724770f027e10c51f56ee9"}
Sep 10 19:55:44.863511 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:44.863384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z7mxd" event={"ID":"4433866a-197a-4fdf-abc5-6c575d5eea05","Type":"ContainerStarted","Data":"d0c25dd38d927a0d0ca2c10467ca5aaedfb25c71c50fa2f27ad973acda3ba43e"}
Sep 10 19:55:45.910052 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:45.910000    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:45.947834 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:45.947802    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"]
Sep 10 19:55:45.950827 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:45.950798    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"
Sep 10 19:55:45.953827 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:45.953802    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 10 19:55:45.953962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:45.953856    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qbkx5\""
Sep 10 19:55:45.953962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:45.953883    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 10 19:55:45.954940 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:45.954919    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 19:55:45.965789 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:45.965764    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"]
Sep 10 19:55:46.034219 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.034179    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:46.132709 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.132680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e777acf3-d8a1-42ca-88ef-5c1cda860bdd-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-dvhwx\" (UID: \"e777acf3-d8a1-42ca-88ef-5c1cda860bdd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"
Sep 10 19:55:46.132868 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.132736    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e777acf3-d8a1-42ca-88ef-5c1cda860bdd-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-dvhwx\" (UID: \"e777acf3-d8a1-42ca-88ef-5c1cda860bdd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"
Sep 10 19:55:46.132868 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.132785    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56ggx\" (UniqueName: \"kubernetes.io/projected/e777acf3-d8a1-42ca-88ef-5c1cda860bdd-kube-api-access-56ggx\") pod \"prometheus-operator-6f4946db56-dvhwx\" (UID: \"e777acf3-d8a1-42ca-88ef-5c1cda860bdd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"
Sep 10 19:55:46.132868 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.132818    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e777acf3-d8a1-42ca-88ef-5c1cda860bdd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-dvhwx\" (UID: \"e777acf3-d8a1-42ca-88ef-5c1cda860bdd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"
Sep 10 19:55:46.216493 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.216468    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:46.233630 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.233601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e777acf3-d8a1-42ca-88ef-5c1cda860bdd-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-dvhwx\" (UID: \"e777acf3-d8a1-42ca-88ef-5c1cda860bdd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"
Sep 10 19:55:46.233729 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.233665    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e777acf3-d8a1-42ca-88ef-5c1cda860bdd-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-dvhwx\" (UID: \"e777acf3-d8a1-42ca-88ef-5c1cda860bdd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"
Sep 10 19:55:46.233729 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.233690    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56ggx\" (UniqueName: \"kubernetes.io/projected/e777acf3-d8a1-42ca-88ef-5c1cda860bdd-kube-api-access-56ggx\") pod \"prometheus-operator-6f4946db56-dvhwx\" (UID: \"e777acf3-d8a1-42ca-88ef-5c1cda860bdd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"
Sep 10 19:55:46.233729 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.233721    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e777acf3-d8a1-42ca-88ef-5c1cda860bdd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-dvhwx\" (UID: \"e777acf3-d8a1-42ca-88ef-5c1cda860bdd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"
Sep 10 19:55:46.234256 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.234237    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e777acf3-d8a1-42ca-88ef-5c1cda860bdd-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-dvhwx\" (UID: \"e777acf3-d8a1-42ca-88ef-5c1cda860bdd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"
Sep 10 19:55:46.236058 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.236035    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e777acf3-d8a1-42ca-88ef-5c1cda860bdd-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-dvhwx\" (UID: \"e777acf3-d8a1-42ca-88ef-5c1cda860bdd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"
Sep 10 19:55:46.236134 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.236073    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e777acf3-d8a1-42ca-88ef-5c1cda860bdd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-dvhwx\" (UID: \"e777acf3-d8a1-42ca-88ef-5c1cda860bdd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"
Sep 10 19:55:46.242230 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.242209    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56ggx\" (UniqueName: \"kubernetes.io/projected/e777acf3-d8a1-42ca-88ef-5c1cda860bdd-kube-api-access-56ggx\") pod \"prometheus-operator-6f4946db56-dvhwx\" (UID: \"e777acf3-d8a1-42ca-88ef-5c1cda860bdd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"
Sep 10 19:55:46.261472 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.261448    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"
Sep 10 19:55:46.534117 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.534086    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-dvhwx"]
Sep 10 19:55:46.537349 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:55:46.537325    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode777acf3_d8a1_42ca_88ef_5c1cda860bdd.slice/crio-fb90180e3ad3ed43eff5c4a50192d337084b54d5f44b7f4f59217ddbf26414aa WatchSource:0}: Error finding container fb90180e3ad3ed43eff5c4a50192d337084b54d5f44b7f4f59217ddbf26414aa: Status 404 returned error can't find the container with id fb90180e3ad3ed43eff5c4a50192d337084b54d5f44b7f4f59217ddbf26414aa
Sep 10 19:55:46.717021 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.716989    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:46.872907 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.872801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z7mxd" event={"ID":"4433866a-197a-4fdf-abc5-6c575d5eea05","Type":"ContainerStarted","Data":"c277b2e84bbd84b954a8eb91bd53d5dab4695baea447b3b5f54e201cd42f4c25"}
Sep 10 19:55:46.874166 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.874147    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx" event={"ID":"e777acf3-d8a1-42ca-88ef-5c1cda860bdd","Type":"ContainerStarted","Data":"fb90180e3ad3ed43eff5c4a50192d337084b54d5f44b7f4f59217ddbf26414aa"}
Sep 10 19:55:46.891048 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:46.891005    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-z7mxd" podStartSLOduration=33.749479097 podStartE2EDuration="35.89098964s" totalImagesPullingTime="2.141510543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:55:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:46.889968944 +0000 UTC m=+87.940496935" watchObservedRunningTime="2026-09-10 19:55:46.89098964 +0000 UTC m=+87.941517626"
Sep 10 19:55:47.770900 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:47.770868    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:55:48.157882 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:48.157803    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:48.541927 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:48.541887    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:48.596492 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:48.596459    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:48.883659 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:48.883583    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx" event={"ID":"e777acf3-d8a1-42ca-88ef-5c1cda860bdd","Type":"ContainerStarted","Data":"f770c6472860b7bf824fdbf6864928cb08cb292d6fff32351361f4f970c74687"}
Sep 10 19:55:48.883659 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:48.883640    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx" event={"ID":"e777acf3-d8a1-42ca-88ef-5c1cda860bdd","Type":"ContainerStarted","Data":"09c0af8bb0c5c3348f2b0f9f75f442fa9a24d6953667609a2be60f8d081c5322"}
Sep 10 19:55:48.904846 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:48.904792    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-dvhwx" podStartSLOduration=2.564512175 podStartE2EDuration="3.904770761s" totalImagesPullingTime="1.340258586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:48.904322328 +0000 UTC m=+89.954850327" watchObservedRunningTime="2026-09-10 19:55:48.904770761 +0000 UTC m=+89.955298749"
Sep 10 19:55:51.065594 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.065547    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:51.212264 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.212213    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:51.321024 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.320912    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:51.332149 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.332119    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kg9km"]
Sep 10 19:55:51.335672 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.335653    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.338188 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.338163    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 19:55:51.338378 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.338343    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 19:55:51.338597 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.338580    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-d8qrq\""
Sep 10 19:55:51.338801 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.338787    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 19:55:51.421624 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.421591    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:51.478682 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.478646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-textfile\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.478868 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.478734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-accelerators-collector-config\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.478868 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.478759    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2549669-329a-42c7-bcd9-72f7430bb367-metrics-client-ca\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.478868 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.478776    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.478868 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.478795    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kz88\" (UniqueName: \"kubernetes.io/projected/f2549669-329a-42c7-bcd9-72f7430bb367-kube-api-access-9kz88\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.478868 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.478855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2549669-329a-42c7-bcd9-72f7430bb367-sys\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.479057 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.478879    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-wtmp\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.479057 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.478901    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-tls\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.479057 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.478926    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2549669-329a-42c7-bcd9-72f7430bb367-root\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.520292 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.520259    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:51.579337 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.579263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2549669-329a-42c7-bcd9-72f7430bb367-sys\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.579337 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.579300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-wtmp\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.579337 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.579329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-tls\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.579572 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.579375    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2549669-329a-42c7-bcd9-72f7430bb367-sys\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.579572 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.579416    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2549669-329a-42c7-bcd9-72f7430bb367-root\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.579572 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.579454    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-textfile\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.579572 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.579469    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-wtmp\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.579572 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.579551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-accelerators-collector-config\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.579572 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.579560    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2549669-329a-42c7-bcd9-72f7430bb367-root\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.579872 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.579589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2549669-329a-42c7-bcd9-72f7430bb367-metrics-client-ca\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.579872 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.579613    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.579872 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.579728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kz88\" (UniqueName: \"kubernetes.io/projected/f2549669-329a-42c7-bcd9-72f7430bb367-kube-api-access-9kz88\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.579872 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.579861    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-textfile\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.580255 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.580230    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-accelerators-collector-config\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.580255 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.580251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2549669-329a-42c7-bcd9-72f7430bb367-metrics-client-ca\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.582238 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.582214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.582238 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.582231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2549669-329a-42c7-bcd9-72f7430bb367-node-exporter-tls\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.595459 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.595430    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kz88\" (UniqueName: \"kubernetes.io/projected/f2549669-329a-42c7-bcd9-72f7430bb367-kube-api-access-9kz88\") pod \"node-exporter-kg9km\" (UID: \"f2549669-329a-42c7-bcd9-72f7430bb367\") " pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.609994 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.609961    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:51.645018 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.644981    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kg9km"
Sep 10 19:55:51.657179 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:55:51.657137    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2549669_329a_42c7_bcd9_72f7430bb367.slice/crio-0fe5b61f8464419a644028c74c1685d066c087467c5369f35ca443f79b5294be WatchSource:0}: Error finding container 0fe5b61f8464419a644028c74c1685d066c087467c5369f35ca443f79b5294be: Status 404 returned error can't find the container with id 0fe5b61f8464419a644028c74c1685d066c087467c5369f35ca443f79b5294be
Sep 10 19:55:51.679178 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.679146    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:51.779553 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.779496    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:51.879496 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.879398    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:51.896376 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:51.896344    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kg9km" event={"ID":"f2549669-329a-42c7-bcd9-72f7430bb367","Type":"ContainerStarted","Data":"0fe5b61f8464419a644028c74c1685d066c087467c5369f35ca443f79b5294be"}
Sep 10 19:55:52.427433 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:52.427370    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:52.525224 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:52.525192    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:52.901471 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:52.901435    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kg9km" event={"ID":"f2549669-329a-42c7-bcd9-72f7430bb367","Type":"ContainerStarted","Data":"bb590e5e42a5c2d2c98baed27d94009c153538283f759792c2d152afa6334e14"}
Sep 10 19:55:53.018298 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.018265    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:53.440660 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.440630    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-864dd47945-42tdf"]
Sep 10 19:55:53.444162 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.444143    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.446944 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.446898    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 19:55:53.447059 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.446945    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 19:55:53.447059 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.447008    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-846cv\""
Sep 10 19:55:53.447159 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.447015    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e5lfibqpc26ih\""
Sep 10 19:55:53.447222 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.447196    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 19:55:53.447267 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.447233    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 19:55:53.447267 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.447242    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 19:55:53.462965 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.462941    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-864dd47945-42tdf"]
Sep 10 19:55:53.495494 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.495455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-tls\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.495494 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.495494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-grpc-tls\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.495793 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.495516    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.495793 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.495602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.495793 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.495636    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.495793 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.495653    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vbxr\" (UniqueName: \"kubernetes.io/projected/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-kube-api-access-2vbxr\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.495793 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.495679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-metrics-client-ca\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.495985 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.495787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.596913 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.596858    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-metrics-client-ca\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.597111 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.596932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.597111 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.596992    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-tls\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.597111 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.597018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-grpc-tls\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.597111 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.597044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.597111 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.597083    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.597111 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.597105    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.597474 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.597128    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vbxr\" (UniqueName: \"kubernetes.io/projected/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-kube-api-access-2vbxr\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.597826 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.597724    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-metrics-client-ca\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.599685 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.599642    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-grpc-tls\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.599877 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.599854    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-tls\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.600038 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.600018    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.600298 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.600278    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.600336 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.600293    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.600474 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.600460    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.612920 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.612887    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vbxr\" (UniqueName: \"kubernetes.io/projected/78da9b4e-ffd2-43ad-b8d7-95f3421aa71a-kube-api-access-2vbxr\") pod \"thanos-querier-864dd47945-42tdf\" (UID: \"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a\") " pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.752808 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.752715    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:53.906828 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.906797    2561 generic.go:356] "Generic (PLEG): container finished" podID="f2549669-329a-42c7-bcd9-72f7430bb367" containerID="bb590e5e42a5c2d2c98baed27d94009c153538283f759792c2d152afa6334e14" exitCode=0
Sep 10 19:55:53.906990 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:53.906835    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kg9km" event={"ID":"f2549669-329a-42c7-bcd9-72f7430bb367","Type":"ContainerDied","Data":"bb590e5e42a5c2d2c98baed27d94009c153538283f759792c2d152afa6334e14"}
Sep 10 19:55:54.028636 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:54.028606    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-864dd47945-42tdf"]
Sep 10 19:55:54.031435 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:55:54.031408    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78da9b4e_ffd2_43ad_b8d7_95f3421aa71a.slice/crio-17f13e35170462d17dbe35fa7f4f6e7c6547b928976b6c43b0eca66391cace98 WatchSource:0}: Error finding container 17f13e35170462d17dbe35fa7f4f6e7c6547b928976b6c43b0eca66391cace98: Status 404 returned error can't find the container with id 17f13e35170462d17dbe35fa7f4f6e7c6547b928976b6c43b0eca66391cace98
Sep 10 19:55:54.303481 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:54.303451    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:54.404015 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:54.403984    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:54.503685 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:54.503648    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:54.856688 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:54.856657    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:54.913677 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:54.913637    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kg9km" event={"ID":"f2549669-329a-42c7-bcd9-72f7430bb367","Type":"ContainerStarted","Data":"b430e470a60cac01ead398b4e7bc54c11609ef18dca8c00d5cba8bedec0d4720"}
Sep 10 19:55:54.913845 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:54.913701    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kg9km" event={"ID":"f2549669-329a-42c7-bcd9-72f7430bb367","Type":"ContainerStarted","Data":"1aee51f14ad9d8542f601a0622556cc97c5502e06d4622c0ebb1b7216f16bb3d"}
Sep 10 19:55:54.915374 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:54.915345    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-864dd47945-42tdf" event={"ID":"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a","Type":"ContainerStarted","Data":"17f13e35170462d17dbe35fa7f4f6e7c6547b928976b6c43b0eca66391cace98"}
Sep 10 19:55:54.937154 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:54.937094    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kg9km" podStartSLOduration=2.729559977 podStartE2EDuration="3.937079087s" totalImagesPullingTime="1.135554704s" totalInitContainerRuntime="71.964406ms" isStatefulPod=true podCreationTimestamp="2026-09-10 19:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:54.936212238 +0000 UTC m=+95.986740240" watchObservedRunningTime="2026-09-10 19:55:54.937079087 +0000 UTC m=+95.987607075"
Sep 10 19:55:55.063171 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:55.063138    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:55.791100 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:55.791067    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:55.852014 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:55.851984    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:56.017437 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:56.017333    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:56.109263 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:56.109219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:56.174772 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:56.174733    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:56.367626 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:56.367552    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:56.930071 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:56.930034    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-864dd47945-42tdf" event={"ID":"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a","Type":"ContainerStarted","Data":"fc67ea4854a535e65fd173274f201a6179f46b355a19e048b751f9839af83bc1"}
Sep 10 19:55:56.930419 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:56.930128    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-864dd47945-42tdf" event={"ID":"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a","Type":"ContainerStarted","Data":"5585a5c471e369018288b27b40dc52f8fb8fe548d9aeaeda90c72b48d2a31ef6"}
Sep 10 19:55:56.930419 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:56.930157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-864dd47945-42tdf" event={"ID":"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a","Type":"ContainerStarted","Data":"158cd4eebae773e1e1198d173dac43048748f64e2aa0fccfd8bb18810381cf74"}
Sep 10 19:55:57.589337 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:57.589304    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:57.737953 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:57.737850    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:57.839453 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:57.839368    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:57.938450 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:57.938407    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-864dd47945-42tdf" event={"ID":"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a","Type":"ContainerStarted","Data":"1545a08989a0f5ee5f3007484f369ad60db5b648ecfedb1cd30401e1b8a8d374"}
Sep 10 19:55:57.938936 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:57.938500    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-864dd47945-42tdf" event={"ID":"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a","Type":"ContainerStarted","Data":"2d68d45732b14e8a24c5c4aee8087475edc0d5d5eabfe6d0f56ac5e03b6dd391"}
Sep 10 19:55:57.938936 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:57.938555    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-864dd47945-42tdf" event={"ID":"78da9b4e-ffd2-43ad-b8d7-95f3421aa71a","Type":"ContainerStarted","Data":"afef42b85fe75f1fe5737b6bae6d1199e2f3bc6901d2564d99ffbed0b4dfed1e"}
Sep 10 19:55:57.939255 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:57.939219    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:55:57.965771 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:57.965699    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-864dd47945-42tdf" podStartSLOduration=2.241059797 podStartE2EDuration="4.965682378s" totalImagesPullingTime="2.724622581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:55:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:55:57.964454081 +0000 UTC m=+99.014982070" watchObservedRunningTime="2026-09-10 19:55:57.965682378 +0000 UTC m=+99.016210367"
Sep 10 19:55:59.233431 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:59.233384    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:59.309986 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:59.309952    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:59.621882 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:59.621847    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:55:59.839945 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:59.839918    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-577c58467f-zwvr7"
Sep 10 19:55:59.874333 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:55:59.874251    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:00.058584 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:00.058544    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:00.162166 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:00.162087    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:00.260110 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:00.260078    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:01.806778 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:01.806751    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lx84z"
Sep 10 19:56:02.782816 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:02.782766    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-556759f-cdwrm" podUID="078c47ec-76a8-41ba-9e75-d6ef705c38ec" containerName="registry" containerID="cri-o://bdf118f8ac69dbecc50d9d3c0a15f1777b1a48f0934bdafffe969a44d6707cbf" gracePeriod=30
Sep 10 19:56:02.958145 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:02.958117    2561 generic.go:356] "Generic (PLEG): container finished" podID="078c47ec-76a8-41ba-9e75-d6ef705c38ec" containerID="bdf118f8ac69dbecc50d9d3c0a15f1777b1a48f0934bdafffe969a44d6707cbf" exitCode=0
Sep 10 19:56:02.958494 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:02.958186    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-556759f-cdwrm" event={"ID":"078c47ec-76a8-41ba-9e75-d6ef705c38ec","Type":"ContainerDied","Data":"bdf118f8ac69dbecc50d9d3c0a15f1777b1a48f0934bdafffe969a44d6707cbf"}
Sep 10 19:56:03.032359 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.032338    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:56:03.077199 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.077126    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/078c47ec-76a8-41ba-9e75-d6ef705c38ec-ca-trust-extracted\") pod \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") "
Sep 10 19:56:03.077199 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.077169    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/078c47ec-76a8-41ba-9e75-d6ef705c38ec-trusted-ca\") pod \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") "
Sep 10 19:56:03.077408 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.077213    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/078c47ec-76a8-41ba-9e75-d6ef705c38ec-image-registry-private-configuration\") pod \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") "
Sep 10 19:56:03.077408 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.077231    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hsqd\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-kube-api-access-7hsqd\") pod \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") "
Sep 10 19:56:03.077408 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.077251    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-certificates\") pod \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") "
Sep 10 19:56:03.077408 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.077268    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/078c47ec-76a8-41ba-9e75-d6ef705c38ec-installation-pull-secrets\") pod \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") "
Sep 10 19:56:03.077408 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.077286    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-bound-sa-token\") pod \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") "
Sep 10 19:56:03.077408 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.077311    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls\") pod \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\" (UID: \"078c47ec-76a8-41ba-9e75-d6ef705c38ec\") "
Sep 10 19:56:03.077823 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.077747    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "078c47ec-76a8-41ba-9e75-d6ef705c38ec" (UID: "078c47ec-76a8-41ba-9e75-d6ef705c38ec"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:56:03.077823 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.077763    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078c47ec-76a8-41ba-9e75-d6ef705c38ec-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "078c47ec-76a8-41ba-9e75-d6ef705c38ec" (UID: "078c47ec-76a8-41ba-9e75-d6ef705c38ec"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:56:03.080116 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.080086    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "078c47ec-76a8-41ba-9e75-d6ef705c38ec" (UID: "078c47ec-76a8-41ba-9e75-d6ef705c38ec"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:56:03.080237 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.080181    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-kube-api-access-7hsqd" (OuterVolumeSpecName: "kube-api-access-7hsqd") pod "078c47ec-76a8-41ba-9e75-d6ef705c38ec" (UID: "078c47ec-76a8-41ba-9e75-d6ef705c38ec"). InnerVolumeSpecName "kube-api-access-7hsqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:56:03.080237 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.080192    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078c47ec-76a8-41ba-9e75-d6ef705c38ec-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "078c47ec-76a8-41ba-9e75-d6ef705c38ec" (UID: "078c47ec-76a8-41ba-9e75-d6ef705c38ec"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:56:03.080356 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.080265    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078c47ec-76a8-41ba-9e75-d6ef705c38ec-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "078c47ec-76a8-41ba-9e75-d6ef705c38ec" (UID: "078c47ec-76a8-41ba-9e75-d6ef705c38ec"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:56:03.080400 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.080357    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "078c47ec-76a8-41ba-9e75-d6ef705c38ec" (UID: "078c47ec-76a8-41ba-9e75-d6ef705c38ec"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:56:03.085858 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.085837    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078c47ec-76a8-41ba-9e75-d6ef705c38ec-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "078c47ec-76a8-41ba-9e75-d6ef705c38ec" (UID: "078c47ec-76a8-41ba-9e75-d6ef705c38ec"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:56:03.178607 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.178567    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/078c47ec-76a8-41ba-9e75-d6ef705c38ec-ca-trust-extracted\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:56:03.178607 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.178598    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/078c47ec-76a8-41ba-9e75-d6ef705c38ec-trusted-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:56:03.178607 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.178608    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/078c47ec-76a8-41ba-9e75-d6ef705c38ec-image-registry-private-configuration\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:56:03.178607 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.178617    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hsqd\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-kube-api-access-7hsqd\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:56:03.178865 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.178625    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-certificates\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:56:03.178865 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.178635    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/078c47ec-76a8-41ba-9e75-d6ef705c38ec-installation-pull-secrets\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:56:03.178865 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.178645    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-bound-sa-token\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:56:03.178865 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.178653    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/078c47ec-76a8-41ba-9e75-d6ef705c38ec-registry-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:56:03.962916 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.962882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-556759f-cdwrm" event={"ID":"078c47ec-76a8-41ba-9e75-d6ef705c38ec","Type":"ContainerDied","Data":"522edba3fa2a45d07ab0861bec968672e709f68405792ef46d83a880a3341a52"}
Sep 10 19:56:03.963355 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.962942    2561 scope.go:117] "RemoveContainer" containerID="bdf118f8ac69dbecc50d9d3c0a15f1777b1a48f0934bdafffe969a44d6707cbf"
Sep 10 19:56:03.963355 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.962894    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-556759f-cdwrm"
Sep 10 19:56:03.980863 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.980839    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-556759f-cdwrm"]
Sep 10 19:56:03.984747 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:03.984727    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-556759f-cdwrm"]
Sep 10 19:56:04.085188 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:04.085156    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:04.145635 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:04.145598    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:04.951750 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:04.951720    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-864dd47945-42tdf"
Sep 10 19:56:05.402279 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:05.402243    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="078c47ec-76a8-41ba-9e75-d6ef705c38ec" path="/var/lib/kubelet/pods/078c47ec-76a8-41ba-9e75-d6ef705c38ec/volumes"
Sep 10 19:56:06.499682 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:06.499650    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:06.556822 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:06.556782    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:08.069173 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:08.069141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:08.181140 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:08.181114    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:11.153585 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:11.153507    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:11.257330 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:11.257292    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:12.648000 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:12.647964    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:12.745858 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:12.745829    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:23.060238 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:23.060200    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:23.158686 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:23.158653    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:28.598425 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:28.598389    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:28.703674 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:28.703645    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:29.303665 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:29.303616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs\") pod \"network-metrics-daemon-cfdm7\" (UID: \"9fce784d-e661-4e90-b91c-49dac8b1e3de\") " pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:56:29.306043 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:29.306020    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fce784d-e661-4e90-b91c-49dac8b1e3de-metrics-certs\") pod \"network-metrics-daemon-cfdm7\" (UID: \"9fce784d-e661-4e90-b91c-49dac8b1e3de\") " pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:56:29.447721 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:29.447685    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mr4n2\""
Sep 10 19:56:29.455979 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:29.455957    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfdm7"
Sep 10 19:56:29.736357 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:29.736283    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cfdm7"]
Sep 10 19:56:29.739235 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:56:29.739206    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fce784d_e661_4e90_b91c_49dac8b1e3de.slice/crio-25c2d1edacfed1e879f5c39aabb3653db32755d94b8d9701634e3a6defe16437 WatchSource:0}: Error finding container 25c2d1edacfed1e879f5c39aabb3653db32755d94b8d9701634e3a6defe16437: Status 404 returned error can't find the container with id 25c2d1edacfed1e879f5c39aabb3653db32755d94b8d9701634e3a6defe16437
Sep 10 19:56:30.065546 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:30.065501    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cfdm7" event={"ID":"9fce784d-e661-4e90-b91c-49dac8b1e3de","Type":"ContainerStarted","Data":"25c2d1edacfed1e879f5c39aabb3653db32755d94b8d9701634e3a6defe16437"}
Sep 10 19:56:31.073073 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:31.073044    2561 generic.go:356] "Generic (PLEG): container finished" podID="7c8de1db-4a70-469e-8f56-fecd54dde30a" containerID="d7220ff41278b079c972b2dcb950526fa7056296027eaa8c4e653aeaeeb795bf" exitCode=0
Sep 10 19:56:31.073504 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:31.073127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm" event={"ID":"7c8de1db-4a70-469e-8f56-fecd54dde30a","Type":"ContainerDied","Data":"d7220ff41278b079c972b2dcb950526fa7056296027eaa8c4e653aeaeeb795bf"}
Sep 10 19:56:31.073504 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:31.073469    2561 scope.go:117] "RemoveContainer" containerID="d7220ff41278b079c972b2dcb950526fa7056296027eaa8c4e653aeaeeb795bf"
Sep 10 19:56:31.075279 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:31.075249    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cfdm7" event={"ID":"9fce784d-e661-4e90-b91c-49dac8b1e3de","Type":"ContainerStarted","Data":"1ccd14f88723365645524e79e4f46f3a5ccec0a47e25c4b2732ed8b5cc5e6706"}
Sep 10 19:56:31.075384 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:31.075299    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cfdm7" event={"ID":"9fce784d-e661-4e90-b91c-49dac8b1e3de","Type":"ContainerStarted","Data":"11342fe908ce1b9e7bb6ab9257b8335e341c9690c250dbd8cb1c5ed3a67c65f4"}
Sep 10 19:56:31.109878 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:31.109825    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cfdm7" podStartSLOduration=131.128388715 podStartE2EDuration="2m12.109808023s" totalImagesPullingTime="981.419308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:56:31.109373717 +0000 UTC m=+132.159901708" watchObservedRunningTime="2026-09-10 19:56:31.109808023 +0000 UTC m=+132.160336010"
Sep 10 19:56:31.228445 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:31.228414    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:31.293165 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:31.293129    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:32.085390 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:32.085352    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vnjtm" event={"ID":"7c8de1db-4a70-469e-8f56-fecd54dde30a","Type":"ContainerStarted","Data":"f9595428bde56583bfc25d39c67ad17a832492bffc4aec8f438d8314baf811bc"}
Sep 10 19:56:38.065095 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:38.065057    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:56:38.164789 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:56:38.164740    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:10.616932 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:10.616842    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:11.628538 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:11.628471    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:18.928734 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:18.928698    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:19.018628 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:19.018595    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:25.901830 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:25.901796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:26.013598 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:26.013563    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:29.940417 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:29.940383    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:36.964414 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:36.964378    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:37.091231 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:37.091202    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:37.185094 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:37.185053    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:37.947334 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:37.947305    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:38.025635 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:38.025606    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:42.522061 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:42.522017    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:53.075884 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:53.075853    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:57:53.174788 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:57:53.174754    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:02.216543 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:02.216491    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:02.350948 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:02.350910    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:12.779348 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:12.779317    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:12.889729 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:12.889702    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:13.587477 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.587439    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"]
Sep 10 19:58:13.587803 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.587789    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="078c47ec-76a8-41ba-9e75-d6ef705c38ec" containerName="registry"
Sep 10 19:58:13.587848 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.587805    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="078c47ec-76a8-41ba-9e75-d6ef705c38ec" containerName="registry"
Sep 10 19:58:13.587885 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.587881    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="078c47ec-76a8-41ba-9e75-d6ef705c38ec" containerName="registry"
Sep 10 19:58:13.591055 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.591039    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"
Sep 10 19:58:13.593410 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.593368    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 19:58:13.593410 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.593397    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-knjsq\""
Sep 10 19:58:13.593616 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.593416    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 19:58:13.598123 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.598103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz\" (UID: \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"
Sep 10 19:58:13.598197 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.598136    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzxvx\" (UniqueName: \"kubernetes.io/projected/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-kube-api-access-lzxvx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz\" (UID: \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"
Sep 10 19:58:13.598243 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.598196    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz\" (UID: \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"
Sep 10 19:58:13.603552 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.603499    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"]
Sep 10 19:58:13.699443 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.699401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz\" (UID: \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"
Sep 10 19:58:13.699443 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.699442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzxvx\" (UniqueName: \"kubernetes.io/projected/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-kube-api-access-lzxvx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz\" (UID: \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"
Sep 10 19:58:13.699683 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.699495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz\" (UID: \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"
Sep 10 19:58:13.704274 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.700275    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz\" (UID: \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"
Sep 10 19:58:13.704274 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.700367    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz\" (UID: \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"
Sep 10 19:58:13.708436 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.708410    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzxvx\" (UniqueName: \"kubernetes.io/projected/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-kube-api-access-lzxvx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz\" (UID: \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"
Sep 10 19:58:13.899704 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:13.899616    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"
Sep 10 19:58:14.175082 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:14.175050    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"]
Sep 10 19:58:14.177826 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:58:14.177791    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee79f960_0345_4cda_bcb5_0a555b2c4cb9.slice/crio-cf23a97ffdc5ec95fe114b26202b3414007314965b3ad8b91a56ed5ae93fb943 WatchSource:0}: Error finding container cf23a97ffdc5ec95fe114b26202b3414007314965b3ad8b91a56ed5ae93fb943: Status 404 returned error can't find the container with id cf23a97ffdc5ec95fe114b26202b3414007314965b3ad8b91a56ed5ae93fb943
Sep 10 19:58:14.454593 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:14.454498    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz" event={"ID":"ee79f960-0345-4cda-bcb5-0a555b2c4cb9","Type":"ContainerStarted","Data":"cf23a97ffdc5ec95fe114b26202b3414007314965b3ad8b91a56ed5ae93fb943"}
Sep 10 19:58:21.488075 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:21.488043    2561 generic.go:356] "Generic (PLEG): container finished" podID="ee79f960-0345-4cda-bcb5-0a555b2c4cb9" containerID="08ea21936deb3608986e19cde71ec865830327a0f190057c2bc18f78f29a66d9" exitCode=0
Sep 10 19:58:21.488482 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:21.488131    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz" event={"ID":"ee79f960-0345-4cda-bcb5-0a555b2c4cb9","Type":"ContainerDied","Data":"08ea21936deb3608986e19cde71ec865830327a0f190057c2bc18f78f29a66d9"}
Sep 10 19:58:24.502018 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:24.501983    2561 generic.go:356] "Generic (PLEG): container finished" podID="ee79f960-0345-4cda-bcb5-0a555b2c4cb9" containerID="0ad3ac8e4a6e293d8cd8511573d2aca34fe0ebae7e960156bc98c68534cc0a92" exitCode=0
Sep 10 19:58:24.502399 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:24.502071    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz" event={"ID":"ee79f960-0345-4cda-bcb5-0a555b2c4cb9","Type":"ContainerDied","Data":"0ad3ac8e4a6e293d8cd8511573d2aca34fe0ebae7e960156bc98c68534cc0a92"}
Sep 10 19:58:31.532514 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:31.532478    2561 generic.go:356] "Generic (PLEG): container finished" podID="ee79f960-0345-4cda-bcb5-0a555b2c4cb9" containerID="0f929b4d236d23501a4e86216e08c20cd0e45e21bb3e2b2d682f96d4ba100ebf" exitCode=0
Sep 10 19:58:31.532960 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:31.532566    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz" event={"ID":"ee79f960-0345-4cda-bcb5-0a555b2c4cb9","Type":"ContainerDied","Data":"0f929b4d236d23501a4e86216e08c20cd0e45e21bb3e2b2d682f96d4ba100ebf"}
Sep 10 19:58:32.658863 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:32.658832    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"
Sep 10 19:58:32.762026 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:32.761986    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-util\") pod \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\" (UID: \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\") "
Sep 10 19:58:32.762214 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:32.762072    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-bundle\") pod \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\" (UID: \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\") "
Sep 10 19:58:32.762214 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:32.762099    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzxvx\" (UniqueName: \"kubernetes.io/projected/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-kube-api-access-lzxvx\") pod \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\" (UID: \"ee79f960-0345-4cda-bcb5-0a555b2c4cb9\") "
Sep 10 19:58:32.762813 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:32.762740    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-bundle" (OuterVolumeSpecName: "bundle") pod "ee79f960-0345-4cda-bcb5-0a555b2c4cb9" (UID: "ee79f960-0345-4cda-bcb5-0a555b2c4cb9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:58:32.764454 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:32.764430    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-kube-api-access-lzxvx" (OuterVolumeSpecName: "kube-api-access-lzxvx") pod "ee79f960-0345-4cda-bcb5-0a555b2c4cb9" (UID: "ee79f960-0345-4cda-bcb5-0a555b2c4cb9"). InnerVolumeSpecName "kube-api-access-lzxvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:58:32.767876 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:32.767852    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-util" (OuterVolumeSpecName: "util") pod "ee79f960-0345-4cda-bcb5-0a555b2c4cb9" (UID: "ee79f960-0345-4cda-bcb5-0a555b2c4cb9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:58:32.863700 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:32.863590    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:58:32.863700 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:32.863646    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:58:32.863700 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:32.863657    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzxvx\" (UniqueName: \"kubernetes.io/projected/ee79f960-0345-4cda-bcb5-0a555b2c4cb9-kube-api-access-lzxvx\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:58:33.542355 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:33.542324    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz"
Sep 10 19:58:33.542355 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:33.542343    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks7kz" event={"ID":"ee79f960-0345-4cda-bcb5-0a555b2c4cb9","Type":"ContainerDied","Data":"cf23a97ffdc5ec95fe114b26202b3414007314965b3ad8b91a56ed5ae93fb943"}
Sep 10 19:58:33.542589 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:33.542391    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf23a97ffdc5ec95fe114b26202b3414007314965b3ad8b91a56ed5ae93fb943"
Sep 10 19:58:34.754030 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:34.753993    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:35.444907 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:35.444839    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:36.591574 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.591541    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5hk48"]
Sep 10 19:58:36.592159 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.591959    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee79f960-0345-4cda-bcb5-0a555b2c4cb9" containerName="util"
Sep 10 19:58:36.592159 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.591975    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee79f960-0345-4cda-bcb5-0a555b2c4cb9" containerName="util"
Sep 10 19:58:36.592159 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.591986    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee79f960-0345-4cda-bcb5-0a555b2c4cb9" containerName="extract"
Sep 10 19:58:36.592159 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.591996    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee79f960-0345-4cda-bcb5-0a555b2c4cb9" containerName="extract"
Sep 10 19:58:36.592159 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.592011    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee79f960-0345-4cda-bcb5-0a555b2c4cb9" containerName="pull"
Sep 10 19:58:36.592159 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.592020    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee79f960-0345-4cda-bcb5-0a555b2c4cb9" containerName="pull"
Sep 10 19:58:36.592159 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.592090    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee79f960-0345-4cda-bcb5-0a555b2c4cb9" containerName="extract"
Sep 10 19:58:36.632726 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.632687    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5hk48"]
Sep 10 19:58:36.632897 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.632811    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5hk48"
Sep 10 19:58:36.635521 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.635498    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 10 19:58:36.635713 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.635544    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 10 19:58:36.635713 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.635678    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-dzrc8\""
Sep 10 19:58:36.693094 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.693054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0734b6fe-6415-427d-8bbf-71d6d6d082bd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5hk48\" (UID: \"0734b6fe-6415-427d-8bbf-71d6d6d082bd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5hk48"
Sep 10 19:58:36.693283 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.693117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmwl\" (UniqueName: \"kubernetes.io/projected/0734b6fe-6415-427d-8bbf-71d6d6d082bd-kube-api-access-lrmwl\") pod \"cert-manager-operator-controller-manager-55c444bf86-5hk48\" (UID: \"0734b6fe-6415-427d-8bbf-71d6d6d082bd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5hk48"
Sep 10 19:58:36.793864 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.793822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0734b6fe-6415-427d-8bbf-71d6d6d082bd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5hk48\" (UID: \"0734b6fe-6415-427d-8bbf-71d6d6d082bd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5hk48"
Sep 10 19:58:36.794034 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.793885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmwl\" (UniqueName: \"kubernetes.io/projected/0734b6fe-6415-427d-8bbf-71d6d6d082bd-kube-api-access-lrmwl\") pod \"cert-manager-operator-controller-manager-55c444bf86-5hk48\" (UID: \"0734b6fe-6415-427d-8bbf-71d6d6d082bd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5hk48"
Sep 10 19:58:36.794226 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.794207    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0734b6fe-6415-427d-8bbf-71d6d6d082bd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5hk48\" (UID: \"0734b6fe-6415-427d-8bbf-71d6d6d082bd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5hk48"
Sep 10 19:58:36.804463 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.804430    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmwl\" (UniqueName: \"kubernetes.io/projected/0734b6fe-6415-427d-8bbf-71d6d6d082bd-kube-api-access-lrmwl\") pod \"cert-manager-operator-controller-manager-55c444bf86-5hk48\" (UID: \"0734b6fe-6415-427d-8bbf-71d6d6d082bd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5hk48"
Sep 10 19:58:36.942576 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:36.942453    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5hk48"
Sep 10 19:58:37.249139 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:37.249047    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5hk48"]
Sep 10 19:58:37.252429 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:58:37.252402    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0734b6fe_6415_427d_8bbf_71d6d6d082bd.slice/crio-4687d2a7a5e2ae633cfaa0617736636b49683a5e74a4afc049528d0a9349f9fd WatchSource:0}: Error finding container 4687d2a7a5e2ae633cfaa0617736636b49683a5e74a4afc049528d0a9349f9fd: Status 404 returned error can't find the container with id 4687d2a7a5e2ae633cfaa0617736636b49683a5e74a4afc049528d0a9349f9fd
Sep 10 19:58:37.558984 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:37.558945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5hk48" event={"ID":"0734b6fe-6415-427d-8bbf-71d6d6d082bd","Type":"ContainerStarted","Data":"4687d2a7a5e2ae633cfaa0617736636b49683a5e74a4afc049528d0a9349f9fd"}
Sep 10 19:58:38.276742 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:38.276703    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:38.384741 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:38.384687    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:40.185151 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:40.185116    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:40.261938 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:40.261904    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:40.572942 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:40.572911    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5hk48" event={"ID":"0734b6fe-6415-427d-8bbf-71d6d6d082bd","Type":"ContainerStarted","Data":"c37508c27fa114e841d291b4beab57a84a03cf7d8a0164a66764e7a4a06159dc"}
Sep 10 19:58:40.607954 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:40.607899    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5hk48" podStartSLOduration=2.187816418 podStartE2EDuration="4.607880371s" totalImagesPullingTime="2.420063953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:58:40.601280085 +0000 UTC m=+261.651808087" watchObservedRunningTime="2026-09-10 19:58:40.607880371 +0000 UTC m=+261.658408359"
Sep 10 19:58:40.771011 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:40.770975    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:40.876310 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:40.876229    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:40.939001 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:40.938961    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wqdg5"]
Sep 10 19:58:40.941216 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:40.941198    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wqdg5"
Sep 10 19:58:40.944211 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:40.944189    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 19:58:40.944825 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:40.944810    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 19:58:40.962737 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:40.962693    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wqdg5"]
Sep 10 19:58:41.030780 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.030746    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfq4\" (UniqueName: \"kubernetes.io/projected/697c330e-205a-4676-8919-36343f0dbe9d-kube-api-access-8mfq4\") pod \"cert-manager-webhook-6486ffd997-wqdg5\" (UID: \"697c330e-205a-4676-8919-36343f0dbe9d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wqdg5"
Sep 10 19:58:41.030968 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.030797    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/697c330e-205a-4676-8919-36343f0dbe9d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wqdg5\" (UID: \"697c330e-205a-4676-8919-36343f0dbe9d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wqdg5"
Sep 10 19:58:41.071265 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.071233    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:41.131844 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.131752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfq4\" (UniqueName: \"kubernetes.io/projected/697c330e-205a-4676-8919-36343f0dbe9d-kube-api-access-8mfq4\") pod \"cert-manager-webhook-6486ffd997-wqdg5\" (UID: \"697c330e-205a-4676-8919-36343f0dbe9d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wqdg5"
Sep 10 19:58:41.131844 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.131810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/697c330e-205a-4676-8919-36343f0dbe9d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wqdg5\" (UID: \"697c330e-205a-4676-8919-36343f0dbe9d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wqdg5"
Sep 10 19:58:41.162313 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.162284    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfq4\" (UniqueName: \"kubernetes.io/projected/697c330e-205a-4676-8919-36343f0dbe9d-kube-api-access-8mfq4\") pod \"cert-manager-webhook-6486ffd997-wqdg5\" (UID: \"697c330e-205a-4676-8919-36343f0dbe9d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wqdg5"
Sep 10 19:58:41.162476 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.162464    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/697c330e-205a-4676-8919-36343f0dbe9d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wqdg5\" (UID: \"697c330e-205a-4676-8919-36343f0dbe9d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wqdg5"
Sep 10 19:58:41.221423 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.221393    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:41.250690 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.250662    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wqdg5"
Sep 10 19:58:41.341413 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.341382    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"]
Sep 10 19:58:41.343695 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.343678    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"
Sep 10 19:58:41.346658 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.346638    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 19:58:41.346962 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.346950    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 19:58:41.347130 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.347113    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-knjsq\""
Sep 10 19:58:41.358320 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.358294    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"]
Sep 10 19:58:41.434280 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.434193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f8a5fa6-d725-4a33-bfd5-235cced56bff-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx\" (UID: \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"
Sep 10 19:58:41.434280 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.434229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbkbv\" (UniqueName: \"kubernetes.io/projected/3f8a5fa6-d725-4a33-bfd5-235cced56bff-kube-api-access-tbkbv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx\" (UID: \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"
Sep 10 19:58:41.434280 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.434260    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f8a5fa6-d725-4a33-bfd5-235cced56bff-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx\" (UID: \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"
Sep 10 19:58:41.535713 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.535668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f8a5fa6-d725-4a33-bfd5-235cced56bff-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx\" (UID: \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"
Sep 10 19:58:41.535713 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.535710    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbkbv\" (UniqueName: \"kubernetes.io/projected/3f8a5fa6-d725-4a33-bfd5-235cced56bff-kube-api-access-tbkbv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx\" (UID: \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"
Sep 10 19:58:41.535990 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.535737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f8a5fa6-d725-4a33-bfd5-235cced56bff-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx\" (UID: \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"
Sep 10 19:58:41.536071 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.536050    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f8a5fa6-d725-4a33-bfd5-235cced56bff-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx\" (UID: \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"
Sep 10 19:58:41.536127 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.536109    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f8a5fa6-d725-4a33-bfd5-235cced56bff-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx\" (UID: \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"
Sep 10 19:58:41.547522 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:58:41.547491    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod697c330e_205a_4676_8919_36343f0dbe9d.slice/crio-6a259625f79384243735ab6f7765b114e711a80300b0bc1d1b4c4ac4e347d355 WatchSource:0}: Error finding container 6a259625f79384243735ab6f7765b114e711a80300b0bc1d1b4c4ac4e347d355: Status 404 returned error can't find the container with id 6a259625f79384243735ab6f7765b114e711a80300b0bc1d1b4c4ac4e347d355
Sep 10 19:58:41.548870 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.548847    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wqdg5"]
Sep 10 19:58:41.572849 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.572821    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbkbv\" (UniqueName: \"kubernetes.io/projected/3f8a5fa6-d725-4a33-bfd5-235cced56bff-kube-api-access-tbkbv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx\" (UID: \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"
Sep 10 19:58:41.578432 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.578397    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wqdg5" event={"ID":"697c330e-205a-4676-8919-36343f0dbe9d","Type":"ContainerStarted","Data":"6a259625f79384243735ab6f7765b114e711a80300b0bc1d1b4c4ac4e347d355"}
Sep 10 19:58:41.653031 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.652995    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"
Sep 10 19:58:41.931142 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:41.931115    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"]
Sep 10 19:58:41.933124 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:58:41.933098    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8a5fa6_d725_4a33_bfd5_235cced56bff.slice/crio-ead09f6698a7890db3409b2d1a23243529be73e7e84840f8ba9669b1203771f0 WatchSource:0}: Error finding container ead09f6698a7890db3409b2d1a23243529be73e7e84840f8ba9669b1203771f0: Status 404 returned error can't find the container with id ead09f6698a7890db3409b2d1a23243529be73e7e84840f8ba9669b1203771f0
Sep 10 19:58:42.585621 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:42.585514    2561 generic.go:356] "Generic (PLEG): container finished" podID="3f8a5fa6-d725-4a33-bfd5-235cced56bff" containerID="4afd3f223e4638b4835459826f1dcaea68e89f902d48339522f9a4f04c2c1a44" exitCode=0
Sep 10 19:58:42.585621 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:42.585605    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx" event={"ID":"3f8a5fa6-d725-4a33-bfd5-235cced56bff","Type":"ContainerDied","Data":"4afd3f223e4638b4835459826f1dcaea68e89f902d48339522f9a4f04c2c1a44"}
Sep 10 19:58:42.586194 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:42.585663    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx" event={"ID":"3f8a5fa6-d725-4a33-bfd5-235cced56bff","Type":"ContainerStarted","Data":"ead09f6698a7890db3409b2d1a23243529be73e7e84840f8ba9669b1203771f0"}
Sep 10 19:58:43.136743 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:43.136706    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:43.324878 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:43.324846    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:44.596631 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:44.596599    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wqdg5" event={"ID":"697c330e-205a-4676-8919-36343f0dbe9d","Type":"ContainerStarted","Data":"5c1ba604239a10b11438930ab5a66ce5943399b5d1f02f63cec09a76f54f8db8"}
Sep 10 19:58:44.597078 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:44.596709    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wqdg5"
Sep 10 19:58:44.618784 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:44.618728    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wqdg5" podStartSLOduration=1.976775459 podStartE2EDuration="4.618709478s" totalImagesPullingTime="2.641934019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:58:44.617509116 +0000 UTC m=+265.668037108" watchObservedRunningTime="2026-09-10 19:58:44.618709478 +0000 UTC m=+265.669237466"
Sep 10 19:58:44.863751 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:44.863676    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:44.933028 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:44.932981    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:46.607801 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:46.607767    2561 generic.go:356] "Generic (PLEG): container finished" podID="3f8a5fa6-d725-4a33-bfd5-235cced56bff" containerID="dd9a6a50a38242487b0da9f308868661a3ba05ee35b3bb13d8d655347c40537a" exitCode=0
Sep 10 19:58:46.608248 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:46.607860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx" event={"ID":"3f8a5fa6-d725-4a33-bfd5-235cced56bff","Type":"ContainerDied","Data":"dd9a6a50a38242487b0da9f308868661a3ba05ee35b3bb13d8d655347c40537a"}
Sep 10 19:58:47.613945 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:47.613913    2561 generic.go:356] "Generic (PLEG): container finished" podID="3f8a5fa6-d725-4a33-bfd5-235cced56bff" containerID="cee0c462b59b3385085f28f93d0abf1f578a0715c6ab49d0227b6ae662c40c7f" exitCode=0
Sep 10 19:58:47.614337 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:47.613998    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx" event={"ID":"3f8a5fa6-d725-4a33-bfd5-235cced56bff","Type":"ContainerDied","Data":"cee0c462b59b3385085f28f93d0abf1f578a0715c6ab49d0227b6ae662c40c7f"}
Sep 10 19:58:48.738660 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:48.738583    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"
Sep 10 19:58:48.896889 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:48.896808    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbkbv\" (UniqueName: \"kubernetes.io/projected/3f8a5fa6-d725-4a33-bfd5-235cced56bff-kube-api-access-tbkbv\") pod \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\" (UID: \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\") "
Sep 10 19:58:48.896889 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:48.896872    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f8a5fa6-d725-4a33-bfd5-235cced56bff-util\") pod \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\" (UID: \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\") "
Sep 10 19:58:48.897080 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:48.896917    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f8a5fa6-d725-4a33-bfd5-235cced56bff-bundle\") pod \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\" (UID: \"3f8a5fa6-d725-4a33-bfd5-235cced56bff\") "
Sep 10 19:58:48.897278 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:48.897257    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8a5fa6-d725-4a33-bfd5-235cced56bff-bundle" (OuterVolumeSpecName: "bundle") pod "3f8a5fa6-d725-4a33-bfd5-235cced56bff" (UID: "3f8a5fa6-d725-4a33-bfd5-235cced56bff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:58:48.899092 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:48.899066    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8a5fa6-d725-4a33-bfd5-235cced56bff-kube-api-access-tbkbv" (OuterVolumeSpecName: "kube-api-access-tbkbv") pod "3f8a5fa6-d725-4a33-bfd5-235cced56bff" (UID: "3f8a5fa6-d725-4a33-bfd5-235cced56bff"). InnerVolumeSpecName "kube-api-access-tbkbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:58:48.901723 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:48.901691    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8a5fa6-d725-4a33-bfd5-235cced56bff-util" (OuterVolumeSpecName: "util") pod "3f8a5fa6-d725-4a33-bfd5-235cced56bff" (UID: "3f8a5fa6-d725-4a33-bfd5-235cced56bff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:58:48.998150 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:48.998112    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbkbv\" (UniqueName: \"kubernetes.io/projected/3f8a5fa6-d725-4a33-bfd5-235cced56bff-kube-api-access-tbkbv\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:58:48.998150 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:48.998149    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f8a5fa6-d725-4a33-bfd5-235cced56bff-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:58:48.998150 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:48.998161    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f8a5fa6-d725-4a33-bfd5-235cced56bff-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:58:49.624999 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:49.624960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx" event={"ID":"3f8a5fa6-d725-4a33-bfd5-235cced56bff","Type":"ContainerDied","Data":"ead09f6698a7890db3409b2d1a23243529be73e7e84840f8ba9669b1203771f0"}
Sep 10 19:58:49.624999 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:49.624987    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjlqlx"
Sep 10 19:58:49.625188 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:49.625022    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ead09f6698a7890db3409b2d1a23243529be73e7e84840f8ba9669b1203771f0"
Sep 10 19:58:50.544271 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:50.544235    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:50.605290 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:50.605263    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wqdg5"
Sep 10 19:58:52.441213 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.441177    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-z2fmn"]
Sep 10 19:58:52.441618 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.441457    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f8a5fa6-d725-4a33-bfd5-235cced56bff" containerName="pull"
Sep 10 19:58:52.441618 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.441469    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f8a5fa6-d725-4a33-bfd5-235cced56bff" containerName="pull"
Sep 10 19:58:52.441618 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.441486    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f8a5fa6-d725-4a33-bfd5-235cced56bff" containerName="util"
Sep 10 19:58:52.441618 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.441492    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f8a5fa6-d725-4a33-bfd5-235cced56bff" containerName="util"
Sep 10 19:58:52.441618 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.441497    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f8a5fa6-d725-4a33-bfd5-235cced56bff" containerName="extract"
Sep 10 19:58:52.441618 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.441502    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f8a5fa6-d725-4a33-bfd5-235cced56bff" containerName="extract"
Sep 10 19:58:52.441618 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.441560    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f8a5fa6-d725-4a33-bfd5-235cced56bff" containerName="extract"
Sep 10 19:58:52.443330 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.443314    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-z2fmn"
Sep 10 19:58:52.445756 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.445734    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-sg6v5\""
Sep 10 19:58:52.454992 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.454968    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-z2fmn"]
Sep 10 19:58:52.524966 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.524923    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdc962ab-e136-4639-9bba-c2431956ff56-bound-sa-token\") pod \"cert-manager-55b4d954d5-z2fmn\" (UID: \"cdc962ab-e136-4639-9bba-c2431956ff56\") " pod="cert-manager/cert-manager-55b4d954d5-z2fmn"
Sep 10 19:58:52.525155 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.524987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsjss\" (UniqueName: \"kubernetes.io/projected/cdc962ab-e136-4639-9bba-c2431956ff56-kube-api-access-xsjss\") pod \"cert-manager-55b4d954d5-z2fmn\" (UID: \"cdc962ab-e136-4639-9bba-c2431956ff56\") " pod="cert-manager/cert-manager-55b4d954d5-z2fmn"
Sep 10 19:58:52.625493 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.625450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdc962ab-e136-4639-9bba-c2431956ff56-bound-sa-token\") pod \"cert-manager-55b4d954d5-z2fmn\" (UID: \"cdc962ab-e136-4639-9bba-c2431956ff56\") " pod="cert-manager/cert-manager-55b4d954d5-z2fmn"
Sep 10 19:58:52.625709 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.625506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsjss\" (UniqueName: \"kubernetes.io/projected/cdc962ab-e136-4639-9bba-c2431956ff56-kube-api-access-xsjss\") pod \"cert-manager-55b4d954d5-z2fmn\" (UID: \"cdc962ab-e136-4639-9bba-c2431956ff56\") " pod="cert-manager/cert-manager-55b4d954d5-z2fmn"
Sep 10 19:58:52.646033 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.646000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdc962ab-e136-4639-9bba-c2431956ff56-bound-sa-token\") pod \"cert-manager-55b4d954d5-z2fmn\" (UID: \"cdc962ab-e136-4639-9bba-c2431956ff56\") " pod="cert-manager/cert-manager-55b4d954d5-z2fmn"
Sep 10 19:58:52.646208 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.646149    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsjss\" (UniqueName: \"kubernetes.io/projected/cdc962ab-e136-4639-9bba-c2431956ff56-kube-api-access-xsjss\") pod \"cert-manager-55b4d954d5-z2fmn\" (UID: \"cdc962ab-e136-4639-9bba-c2431956ff56\") " pod="cert-manager/cert-manager-55b4d954d5-z2fmn"
Sep 10 19:58:52.752086 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:52.751985    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-z2fmn"
Sep 10 19:58:53.027890 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:53.027867    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-z2fmn"]
Sep 10 19:58:53.030039 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:58:53.030010    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdc962ab_e136_4639_9bba_c2431956ff56.slice/crio-08b89c2fa3be1cc87b5d77a6573a19c502cc2f5b9efce254c2145ff32cd466bd WatchSource:0}: Error finding container 08b89c2fa3be1cc87b5d77a6573a19c502cc2f5b9efce254c2145ff32cd466bd: Status 404 returned error can't find the container with id 08b89c2fa3be1cc87b5d77a6573a19c502cc2f5b9efce254c2145ff32cd466bd
Sep 10 19:58:53.643365 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:53.643325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-z2fmn" event={"ID":"cdc962ab-e136-4639-9bba-c2431956ff56","Type":"ContainerStarted","Data":"8064e6d696b1c040934a37a44193a9bdf93f7062bcac50f72b69797d5ae82e9a"}
Sep 10 19:58:53.643365 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:53.643366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-z2fmn" event={"ID":"cdc962ab-e136-4639-9bba-c2431956ff56","Type":"ContainerStarted","Data":"08b89c2fa3be1cc87b5d77a6573a19c502cc2f5b9efce254c2145ff32cd466bd"}
Sep 10 19:58:53.663188 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:53.663132    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-z2fmn" podStartSLOduration=1.663116285 podStartE2EDuration="1.663116285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:58:53.662271979 +0000 UTC m=+274.712799967" watchObservedRunningTime="2026-09-10 19:58:53.663116285 +0000 UTC m=+274.713644271"
Sep 10 19:58:54.275807 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:54.275777    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:54.377597 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:54.377566    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:55.987705 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:55.987669    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8jsw2"]
Sep 10 19:58:55.989961 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:55.989945    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8jsw2"
Sep 10 19:58:55.992696 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:55.992670    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 19:58:55.993000 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:55.992981    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 19:58:55.993000 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:55.993006    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-h8xs6\""
Sep 10 19:58:56.007278 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:56.007251    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8jsw2"]
Sep 10 19:58:56.154975 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:56.154936    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b27fc61-c745-4948-b94e-2a5c1c393a9a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8jsw2\" (UID: \"2b27fc61-c745-4948-b94e-2a5c1c393a9a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8jsw2"
Sep 10 19:58:56.154975 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:56.154976    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89hjt\" (UniqueName: \"kubernetes.io/projected/2b27fc61-c745-4948-b94e-2a5c1c393a9a-kube-api-access-89hjt\") pod \"openshift-lws-operator-fd8ccff4c-8jsw2\" (UID: \"2b27fc61-c745-4948-b94e-2a5c1c393a9a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8jsw2"
Sep 10 19:58:56.256110 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:56.256013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b27fc61-c745-4948-b94e-2a5c1c393a9a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8jsw2\" (UID: \"2b27fc61-c745-4948-b94e-2a5c1c393a9a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8jsw2"
Sep 10 19:58:56.256110 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:56.256059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89hjt\" (UniqueName: \"kubernetes.io/projected/2b27fc61-c745-4948-b94e-2a5c1c393a9a-kube-api-access-89hjt\") pod \"openshift-lws-operator-fd8ccff4c-8jsw2\" (UID: \"2b27fc61-c745-4948-b94e-2a5c1c393a9a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8jsw2"
Sep 10 19:58:56.256339 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:56.256312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b27fc61-c745-4948-b94e-2a5c1c393a9a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8jsw2\" (UID: \"2b27fc61-c745-4948-b94e-2a5c1c393a9a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8jsw2"
Sep 10 19:58:56.265267 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:56.265244    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89hjt\" (UniqueName: \"kubernetes.io/projected/2b27fc61-c745-4948-b94e-2a5c1c393a9a-kube-api-access-89hjt\") pod \"openshift-lws-operator-fd8ccff4c-8jsw2\" (UID: \"2b27fc61-c745-4948-b94e-2a5c1c393a9a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8jsw2"
Sep 10 19:58:56.299300 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:56.299266    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8jsw2"
Sep 10 19:58:56.577672 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:56.577635    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8jsw2"]
Sep 10 19:58:56.580781 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:58:56.580739    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b27fc61_c745_4948_b94e_2a5c1c393a9a.slice/crio-6699befc0ab845ba01099d1ae42c26d3ece90f54b58aff941a12229851688b9b WatchSource:0}: Error finding container 6699befc0ab845ba01099d1ae42c26d3ece90f54b58aff941a12229851688b9b: Status 404 returned error can't find the container with id 6699befc0ab845ba01099d1ae42c26d3ece90f54b58aff941a12229851688b9b
Sep 10 19:58:56.657370 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:56.657328    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8jsw2" event={"ID":"2b27fc61-c745-4948-b94e-2a5c1c393a9a","Type":"ContainerStarted","Data":"6699befc0ab845ba01099d1ae42c26d3ece90f54b58aff941a12229851688b9b"}
Sep 10 19:58:58.667398 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:58.667301    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8jsw2" event={"ID":"2b27fc61-c745-4948-b94e-2a5c1c393a9a","Type":"ContainerStarted","Data":"f151ab3ffb8ab6b20396eb746205e83ccbd0bb775a834045066a69c71eab4515"}
Sep 10 19:58:58.691185 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:58.691133    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8jsw2" podStartSLOduration=1.874044551 podStartE2EDuration="3.691116373s" totalImagesPullingTime="1.817071822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:58:58.688759661 +0000 UTC m=+279.739287650" watchObservedRunningTime="2026-09-10 19:58:58.691116373 +0000 UTC m=+279.741644361"
Sep 10 19:58:59.493208 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:59.493174    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:58:59.591349 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:58:59.591315    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:01.080395 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.080357    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"]
Sep 10 19:59:01.084386 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.084369    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"
Sep 10 19:59:01.087111 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.087092    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 19:59:01.088113 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.088094    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 19:59:01.088217 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.088098    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-knjsq\""
Sep 10 19:59:01.098521 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.098496    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"]
Sep 10 19:59:01.194313 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.194269    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79vsg\" (UniqueName: \"kubernetes.io/projected/b45dc5d1-6684-446a-824c-975dd243305b-kube-api-access-79vsg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk\" (UID: \"b45dc5d1-6684-446a-824c-975dd243305b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"
Sep 10 19:59:01.194313 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.194317    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b45dc5d1-6684-446a-824c-975dd243305b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk\" (UID: \"b45dc5d1-6684-446a-824c-975dd243305b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"
Sep 10 19:59:01.194576 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.194378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b45dc5d1-6684-446a-824c-975dd243305b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk\" (UID: \"b45dc5d1-6684-446a-824c-975dd243305b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"
Sep 10 19:59:01.294895 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.294853    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79vsg\" (UniqueName: \"kubernetes.io/projected/b45dc5d1-6684-446a-824c-975dd243305b-kube-api-access-79vsg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk\" (UID: \"b45dc5d1-6684-446a-824c-975dd243305b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"
Sep 10 19:59:01.294895 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.294901    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b45dc5d1-6684-446a-824c-975dd243305b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk\" (UID: \"b45dc5d1-6684-446a-824c-975dd243305b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"
Sep 10 19:59:01.295160 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.294929    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b45dc5d1-6684-446a-824c-975dd243305b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk\" (UID: \"b45dc5d1-6684-446a-824c-975dd243305b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"
Sep 10 19:59:01.295288 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.295271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b45dc5d1-6684-446a-824c-975dd243305b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk\" (UID: \"b45dc5d1-6684-446a-824c-975dd243305b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"
Sep 10 19:59:01.295360 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.295313    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b45dc5d1-6684-446a-824c-975dd243305b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk\" (UID: \"b45dc5d1-6684-446a-824c-975dd243305b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"
Sep 10 19:59:01.303672 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.303643    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79vsg\" (UniqueName: \"kubernetes.io/projected/b45dc5d1-6684-446a-824c-975dd243305b-kube-api-access-79vsg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk\" (UID: \"b45dc5d1-6684-446a-824c-975dd243305b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"
Sep 10 19:59:01.393678 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.393596    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"
Sep 10 19:59:01.505887 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.505852    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:01.686716 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:01.686687    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"]
Sep 10 19:59:01.688989 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:59:01.688965    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45dc5d1_6684_446a_824c_975dd243305b.slice/crio-98c4ba0a3cfb24dfe638b287aa8d88520afd6912c26ffec0284ba8f4dba9e471 WatchSource:0}: Error finding container 98c4ba0a3cfb24dfe638b287aa8d88520afd6912c26ffec0284ba8f4dba9e471: Status 404 returned error can't find the container with id 98c4ba0a3cfb24dfe638b287aa8d88520afd6912c26ffec0284ba8f4dba9e471
Sep 10 19:59:02.686487 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:02.686399    2561 generic.go:356] "Generic (PLEG): container finished" podID="b45dc5d1-6684-446a-824c-975dd243305b" containerID="266a646ca57cfb52f8957f19d9afbf30030bf73c63ef64dbb9fdaac4c318b07e" exitCode=0
Sep 10 19:59:02.686891 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:02.686489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk" event={"ID":"b45dc5d1-6684-446a-824c-975dd243305b","Type":"ContainerDied","Data":"266a646ca57cfb52f8957f19d9afbf30030bf73c63ef64dbb9fdaac4c318b07e"}
Sep 10 19:59:02.686891 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:02.686559    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk" event={"ID":"b45dc5d1-6684-446a-824c-975dd243305b","Type":"ContainerStarted","Data":"98c4ba0a3cfb24dfe638b287aa8d88520afd6912c26ffec0284ba8f4dba9e471"}
Sep 10 19:59:03.692703 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:03.692618    2561 generic.go:356] "Generic (PLEG): container finished" podID="b45dc5d1-6684-446a-824c-975dd243305b" containerID="e0001b09ad9491f1663a0dcfab7aebd1f685d76a27695c3a1426a363a3bcdc28" exitCode=0
Sep 10 19:59:03.693156 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:03.692710    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk" event={"ID":"b45dc5d1-6684-446a-824c-975dd243305b","Type":"ContainerDied","Data":"e0001b09ad9491f1663a0dcfab7aebd1f685d76a27695c3a1426a363a3bcdc28"}
Sep 10 19:59:04.699536 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:04.699499    2561 generic.go:356] "Generic (PLEG): container finished" podID="b45dc5d1-6684-446a-824c-975dd243305b" containerID="ea0821ec2cf9278f988db10e466e706a74af948d1e82111dd2637ef6b7383e98" exitCode=0
Sep 10 19:59:04.699953 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:04.699561    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk" event={"ID":"b45dc5d1-6684-446a-824c-975dd243305b","Type":"ContainerDied","Data":"ea0821ec2cf9278f988db10e466e706a74af948d1e82111dd2637ef6b7383e98"}
Sep 10 19:59:05.828731 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:05.828704    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"
Sep 10 19:59:05.929947 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:05.929898    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b45dc5d1-6684-446a-824c-975dd243305b-util\") pod \"b45dc5d1-6684-446a-824c-975dd243305b\" (UID: \"b45dc5d1-6684-446a-824c-975dd243305b\") "
Sep 10 19:59:05.930147 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:05.929971    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79vsg\" (UniqueName: \"kubernetes.io/projected/b45dc5d1-6684-446a-824c-975dd243305b-kube-api-access-79vsg\") pod \"b45dc5d1-6684-446a-824c-975dd243305b\" (UID: \"b45dc5d1-6684-446a-824c-975dd243305b\") "
Sep 10 19:59:05.930147 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:05.930008    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b45dc5d1-6684-446a-824c-975dd243305b-bundle\") pod \"b45dc5d1-6684-446a-824c-975dd243305b\" (UID: \"b45dc5d1-6684-446a-824c-975dd243305b\") "
Sep 10 19:59:05.930809 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:05.930732    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45dc5d1-6684-446a-824c-975dd243305b-bundle" (OuterVolumeSpecName: "bundle") pod "b45dc5d1-6684-446a-824c-975dd243305b" (UID: "b45dc5d1-6684-446a-824c-975dd243305b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:59:05.932376 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:05.932350    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45dc5d1-6684-446a-824c-975dd243305b-kube-api-access-79vsg" (OuterVolumeSpecName: "kube-api-access-79vsg") pod "b45dc5d1-6684-446a-824c-975dd243305b" (UID: "b45dc5d1-6684-446a-824c-975dd243305b"). InnerVolumeSpecName "kube-api-access-79vsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:59:05.935592 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:05.935568    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45dc5d1-6684-446a-824c-975dd243305b-util" (OuterVolumeSpecName: "util") pod "b45dc5d1-6684-446a-824c-975dd243305b" (UID: "b45dc5d1-6684-446a-824c-975dd243305b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:59:06.030949 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:06.030914    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b45dc5d1-6684-446a-824c-975dd243305b-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:59:06.030949 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:06.030949    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79vsg\" (UniqueName: \"kubernetes.io/projected/b45dc5d1-6684-446a-824c-975dd243305b-kube-api-access-79vsg\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:59:06.031137 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:06.030963    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b45dc5d1-6684-446a-824c-975dd243305b-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:59:06.709122 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:06.709095    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk"
Sep 10 19:59:06.709312 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:06.709122    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmrbk" event={"ID":"b45dc5d1-6684-446a-824c-975dd243305b","Type":"ContainerDied","Data":"98c4ba0a3cfb24dfe638b287aa8d88520afd6912c26ffec0284ba8f4dba9e471"}
Sep 10 19:59:06.709312 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:06.709178    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98c4ba0a3cfb24dfe638b287aa8d88520afd6912c26ffec0284ba8f4dba9e471"
Sep 10 19:59:14.726261 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.726228    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"]
Sep 10 19:59:14.726767 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.726586    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45dc5d1-6684-446a-824c-975dd243305b" containerName="extract"
Sep 10 19:59:14.726767 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.726600    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45dc5d1-6684-446a-824c-975dd243305b" containerName="extract"
Sep 10 19:59:14.726767 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.726622    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45dc5d1-6684-446a-824c-975dd243305b" containerName="util"
Sep 10 19:59:14.726767 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.726628    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45dc5d1-6684-446a-824c-975dd243305b" containerName="util"
Sep 10 19:59:14.726767 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.726635    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45dc5d1-6684-446a-824c-975dd243305b" containerName="pull"
Sep 10 19:59:14.726767 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.726640    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45dc5d1-6684-446a-824c-975dd243305b" containerName="pull"
Sep 10 19:59:14.726767 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.726684    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b45dc5d1-6684-446a-824c-975dd243305b" containerName="extract"
Sep 10 19:59:14.728897 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.728881    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"
Sep 10 19:59:14.732195 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.732177    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 19:59:14.732318 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.732299    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-knjsq\""
Sep 10 19:59:14.733066 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.733049    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 19:59:14.765361 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.765329    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"]
Sep 10 19:59:14.790663 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.790628    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a83b9734-1e1a-4e38-8ede-573069f9216d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh\" (UID: \"a83b9734-1e1a-4e38-8ede-573069f9216d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"
Sep 10 19:59:14.790823 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.790674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a83b9734-1e1a-4e38-8ede-573069f9216d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh\" (UID: \"a83b9734-1e1a-4e38-8ede-573069f9216d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"
Sep 10 19:59:14.790823 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.790746    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8b46\" (UniqueName: \"kubernetes.io/projected/a83b9734-1e1a-4e38-8ede-573069f9216d-kube-api-access-k8b46\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh\" (UID: \"a83b9734-1e1a-4e38-8ede-573069f9216d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"
Sep 10 19:59:14.891549 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.891483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a83b9734-1e1a-4e38-8ede-573069f9216d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh\" (UID: \"a83b9734-1e1a-4e38-8ede-573069f9216d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"
Sep 10 19:59:14.891732 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.891562    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a83b9734-1e1a-4e38-8ede-573069f9216d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh\" (UID: \"a83b9734-1e1a-4e38-8ede-573069f9216d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"
Sep 10 19:59:14.891732 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.891598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8b46\" (UniqueName: \"kubernetes.io/projected/a83b9734-1e1a-4e38-8ede-573069f9216d-kube-api-access-k8b46\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh\" (UID: \"a83b9734-1e1a-4e38-8ede-573069f9216d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"
Sep 10 19:59:14.891907 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.891884    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a83b9734-1e1a-4e38-8ede-573069f9216d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh\" (UID: \"a83b9734-1e1a-4e38-8ede-573069f9216d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"
Sep 10 19:59:14.891945 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.891909    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a83b9734-1e1a-4e38-8ede-573069f9216d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh\" (UID: \"a83b9734-1e1a-4e38-8ede-573069f9216d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"
Sep 10 19:59:14.905618 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:14.905584    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8b46\" (UniqueName: \"kubernetes.io/projected/a83b9734-1e1a-4e38-8ede-573069f9216d-kube-api-access-k8b46\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh\" (UID: \"a83b9734-1e1a-4e38-8ede-573069f9216d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"
Sep 10 19:59:15.038145 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:15.038115    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"
Sep 10 19:59:15.332069 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:15.331985    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"]
Sep 10 19:59:15.335116 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:59:15.335086    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83b9734_1e1a_4e38_8ede_573069f9216d.slice/crio-e63c8b15067106e4914275ff090ec1ccf03862a08ca9325a997178585401b2e9 WatchSource:0}: Error finding container e63c8b15067106e4914275ff090ec1ccf03862a08ca9325a997178585401b2e9: Status 404 returned error can't find the container with id e63c8b15067106e4914275ff090ec1ccf03862a08ca9325a997178585401b2e9
Sep 10 19:59:15.749120 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:15.749026    2561 generic.go:356] "Generic (PLEG): container finished" podID="a83b9734-1e1a-4e38-8ede-573069f9216d" containerID="7b948bdd051fd42e557264f69999fdfdd069747ca7af943fb778d848a6a93e29" exitCode=0
Sep 10 19:59:15.749120 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:15.749081    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh" event={"ID":"a83b9734-1e1a-4e38-8ede-573069f9216d","Type":"ContainerDied","Data":"7b948bdd051fd42e557264f69999fdfdd069747ca7af943fb778d848a6a93e29"}
Sep 10 19:59:15.749514 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:15.749122    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh" event={"ID":"a83b9734-1e1a-4e38-8ede-573069f9216d","Type":"ContainerStarted","Data":"e63c8b15067106e4914275ff090ec1ccf03862a08ca9325a997178585401b2e9"}
Sep 10 19:59:16.040706 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:16.040677    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:16.148430 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:16.148393    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:16.640167 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:16.640139    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:16.756317 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:16.756219    2561 generic.go:356] "Generic (PLEG): container finished" podID="a83b9734-1e1a-4e38-8ede-573069f9216d" containerID="ded09bdaf9e1d99377958f2b38ad4b84c6b344c44d749b0f94e6d816ef11c083" exitCode=0
Sep 10 19:59:16.756317 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:16.756267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh" event={"ID":"a83b9734-1e1a-4e38-8ede-573069f9216d","Type":"ContainerDied","Data":"ded09bdaf9e1d99377958f2b38ad4b84c6b344c44d749b0f94e6d816ef11c083"}
Sep 10 19:59:16.968723 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:16.968692    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:17.085024 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.084988    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:17.136720 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.136681    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"]
Sep 10 19:59:17.138885 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.138863    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.141939 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.141921    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 19:59:17.142394 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.142381    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 19:59:17.142481 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.142383    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 19:59:17.142642 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.142576    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gxjbr\""
Sep 10 19:59:17.142710 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.142671    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 19:59:17.179932 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.177612    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"]
Sep 10 19:59:17.208304 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.208268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9d14710-c876-4054-9773-93ef56c10bf4-webhook-cert\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.208304 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.208305    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9d14710-c876-4054-9773-93ef56c10bf4-tmp\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.208585 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.208351    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dwpd\" (UniqueName: \"kubernetes.io/projected/b9d14710-c876-4054-9773-93ef56c10bf4-kube-api-access-4dwpd\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.208585 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.208459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9d14710-c876-4054-9773-93ef56c10bf4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.208585 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.208515    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9d14710-c876-4054-9773-93ef56c10bf4-manifests\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.290028 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.289965    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:17.309222 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.309189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dwpd\" (UniqueName: \"kubernetes.io/projected/b9d14710-c876-4054-9773-93ef56c10bf4-kube-api-access-4dwpd\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.309385 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.309259    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9d14710-c876-4054-9773-93ef56c10bf4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.309385 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.309299    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9d14710-c876-4054-9773-93ef56c10bf4-manifests\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.309385 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.309322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9d14710-c876-4054-9773-93ef56c10bf4-webhook-cert\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.309385 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.309349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9d14710-c876-4054-9773-93ef56c10bf4-tmp\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.309727 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.309700    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9d14710-c876-4054-9773-93ef56c10bf4-manifests\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.311876 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.311847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9d14710-c876-4054-9773-93ef56c10bf4-tmp\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.312065 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.312047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9d14710-c876-4054-9773-93ef56c10bf4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.312177 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.312156    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9d14710-c876-4054-9773-93ef56c10bf4-webhook-cert\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.320068 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.320045    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dwpd\" (UniqueName: \"kubernetes.io/projected/b9d14710-c876-4054-9773-93ef56c10bf4-kube-api-access-4dwpd\") pod \"opendatahub-operator-controller-manager-5678c4c4c5-hvgs9\" (UID: \"b9d14710-c876-4054-9773-93ef56c10bf4\") " pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.391750 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.391668    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:17.448868 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.448836    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:17.492836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.492804    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:17.595210 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.595150    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:17.695063 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.694975    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:17.761972 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.761930    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"]
Sep 10 19:59:17.766280 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:59:17.766249    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d14710_c876_4054_9773_93ef56c10bf4.slice/crio-bde0c96fb88a3236952451504321ff65c76d23270d0e59c7cfbefcacff33e834 WatchSource:0}: Error finding container bde0c96fb88a3236952451504321ff65c76d23270d0e59c7cfbefcacff33e834: Status 404 returned error can't find the container with id bde0c96fb88a3236952451504321ff65c76d23270d0e59c7cfbefcacff33e834
Sep 10 19:59:17.766402 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.766339    2561 generic.go:356] "Generic (PLEG): container finished" podID="a83b9734-1e1a-4e38-8ede-573069f9216d" containerID="4fdc65999ad62a3cb02f7b1b95973e0476280c3ccca1f2155d068b81cd338dcc" exitCode=0
Sep 10 19:59:17.766462 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.766427    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh" event={"ID":"a83b9734-1e1a-4e38-8ede-573069f9216d","Type":"ContainerDied","Data":"4fdc65999ad62a3cb02f7b1b95973e0476280c3ccca1f2155d068b81cd338dcc"}
Sep 10 19:59:17.835735 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:17.835697    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:18.774220 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:18.774144    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9" event={"ID":"b9d14710-c876-4054-9773-93ef56c10bf4","Type":"ContainerStarted","Data":"bde0c96fb88a3236952451504321ff65c76d23270d0e59c7cfbefcacff33e834"}
Sep 10 19:59:18.976595 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:18.976563    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"
Sep 10 19:59:19.021059 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:19.021028    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8b46\" (UniqueName: \"kubernetes.io/projected/a83b9734-1e1a-4e38-8ede-573069f9216d-kube-api-access-k8b46\") pod \"a83b9734-1e1a-4e38-8ede-573069f9216d\" (UID: \"a83b9734-1e1a-4e38-8ede-573069f9216d\") "
Sep 10 19:59:19.021238 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:19.021081    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a83b9734-1e1a-4e38-8ede-573069f9216d-util\") pod \"a83b9734-1e1a-4e38-8ede-573069f9216d\" (UID: \"a83b9734-1e1a-4e38-8ede-573069f9216d\") "
Sep 10 19:59:19.021238 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:19.021124    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a83b9734-1e1a-4e38-8ede-573069f9216d-bundle\") pod \"a83b9734-1e1a-4e38-8ede-573069f9216d\" (UID: \"a83b9734-1e1a-4e38-8ede-573069f9216d\") "
Sep 10 19:59:19.022150 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:19.022115    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83b9734-1e1a-4e38-8ede-573069f9216d-bundle" (OuterVolumeSpecName: "bundle") pod "a83b9734-1e1a-4e38-8ede-573069f9216d" (UID: "a83b9734-1e1a-4e38-8ede-573069f9216d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:59:19.023824 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:19.023796    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83b9734-1e1a-4e38-8ede-573069f9216d-kube-api-access-k8b46" (OuterVolumeSpecName: "kube-api-access-k8b46") pod "a83b9734-1e1a-4e38-8ede-573069f9216d" (UID: "a83b9734-1e1a-4e38-8ede-573069f9216d"). InnerVolumeSpecName "kube-api-access-k8b46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:59:19.028663 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:19.028602    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83b9734-1e1a-4e38-8ede-573069f9216d-util" (OuterVolumeSpecName: "util") pod "a83b9734-1e1a-4e38-8ede-573069f9216d" (UID: "a83b9734-1e1a-4e38-8ede-573069f9216d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:59:19.121836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:19.121789    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8b46\" (UniqueName: \"kubernetes.io/projected/a83b9734-1e1a-4e38-8ede-573069f9216d-kube-api-access-k8b46\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:59:19.121836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:19.121826    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a83b9734-1e1a-4e38-8ede-573069f9216d-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:59:19.121836 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:19.121839    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a83b9734-1e1a-4e38-8ede-573069f9216d-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 19:59:19.800489 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:19.780816    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh" event={"ID":"a83b9734-1e1a-4e38-8ede-573069f9216d","Type":"ContainerDied","Data":"e63c8b15067106e4914275ff090ec1ccf03862a08ca9325a997178585401b2e9"}
Sep 10 19:59:19.800489 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:19.780842    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tbpbh"
Sep 10 19:59:19.800489 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:19.780862    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e63c8b15067106e4914275ff090ec1ccf03862a08ca9325a997178585401b2e9"
Sep 10 19:59:19.890729 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:19.890681    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:19.992034 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:19.991992    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:20.734972 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:20.734943    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 19:59:20.734972 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:20.734974    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 19:59:20.745260 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:20.745238    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 19:59:21.790928 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:21.790892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9" event={"ID":"b9d14710-c876-4054-9773-93ef56c10bf4","Type":"ContainerStarted","Data":"17b64019d8ab34a074ad2dac7626412f798aa02bb5861b36d56da88db156ba29"}
Sep 10 19:59:22.797196 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:22.797162    2561 generic.go:356] "Generic (PLEG): container finished" podID="b9d14710-c876-4054-9773-93ef56c10bf4" containerID="17b64019d8ab34a074ad2dac7626412f798aa02bb5861b36d56da88db156ba29" exitCode=0
Sep 10 19:59:22.797196 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:22.797204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9" event={"ID":"b9d14710-c876-4054-9773-93ef56c10bf4","Type":"ContainerDied","Data":"17b64019d8ab34a074ad2dac7626412f798aa02bb5861b36d56da88db156ba29"}
Sep 10 19:59:22.797729 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:22.797715    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 19:59:23.803734 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:23.803700    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9" event={"ID":"b9d14710-c876-4054-9773-93ef56c10bf4","Type":"ContainerStarted","Data":"99b57abeafc5ac86d99267ffc8f3dbeeeda78b454cc2a9b3a3410fccb01a0803"}
Sep 10 19:59:23.804193 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:23.803835    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:23.828399 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:23.828347    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9" podStartSLOduration=2.22406755 podStartE2EDuration="6.828331498s" totalImagesPullingTime="3.58361559s" totalInitContainerRuntime="1.020648358s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:59:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:59:23.826820183 +0000 UTC m=+304.877348184" watchObservedRunningTime="2026-09-10 19:59:23.828331498 +0000 UTC m=+304.878859487"
Sep 10 19:59:25.788407 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:25.788377    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:25.889273 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:25.889241    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:34.810687 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:34.810655    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5678c4c4c5-hvgs9"
Sep 10 19:59:36.355748 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:36.355717    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:37.783760 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:37.783725    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:42.369667 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.369631    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:42.409145 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.409110    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"]
Sep 10 19:59:42.409445 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.409432    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83b9734-1e1a-4e38-8ede-573069f9216d" containerName="util"
Sep 10 19:59:42.409492 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.409448    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83b9734-1e1a-4e38-8ede-573069f9216d" containerName="util"
Sep 10 19:59:42.409492 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.409465    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83b9734-1e1a-4e38-8ede-573069f9216d" containerName="pull"
Sep 10 19:59:42.409492 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.409471    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83b9734-1e1a-4e38-8ede-573069f9216d" containerName="pull"
Sep 10 19:59:42.409492 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.409483    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83b9734-1e1a-4e38-8ede-573069f9216d" containerName="extract"
Sep 10 19:59:42.409492 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.409490    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83b9734-1e1a-4e38-8ede-573069f9216d" containerName="extract"
Sep 10 19:59:42.409663 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.409578    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83b9734-1e1a-4e38-8ede-573069f9216d" containerName="extract"
Sep 10 19:59:42.412482 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.412462    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"
Sep 10 19:59:42.415634 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.415591    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 19:59:42.415734 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.415718    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 19:59:42.415984 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.415968    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 19:59:42.416619 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.416604    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 19:59:42.416704 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.416611    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-srvpp\""
Sep 10 19:59:42.426750 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.426729    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"]
Sep 10 19:59:42.499321 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.499287    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb2c398-ca3f-4378-9eb9-93ce9bac8db5-tls-certs\") pod \"kube-auth-proxy-576c9858ff-hknzf\" (UID: \"7bb2c398-ca3f-4378-9eb9-93ce9bac8db5\") " pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"
Sep 10 19:59:42.499501 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.499369    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bb2c398-ca3f-4378-9eb9-93ce9bac8db5-tmp\") pod \"kube-auth-proxy-576c9858ff-hknzf\" (UID: \"7bb2c398-ca3f-4378-9eb9-93ce9bac8db5\") " pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"
Sep 10 19:59:42.499501 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.499424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrhxf\" (UniqueName: \"kubernetes.io/projected/7bb2c398-ca3f-4378-9eb9-93ce9bac8db5-kube-api-access-qrhxf\") pod \"kube-auth-proxy-576c9858ff-hknzf\" (UID: \"7bb2c398-ca3f-4378-9eb9-93ce9bac8db5\") " pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"
Sep 10 19:59:42.600357 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.600310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb2c398-ca3f-4378-9eb9-93ce9bac8db5-tls-certs\") pod \"kube-auth-proxy-576c9858ff-hknzf\" (UID: \"7bb2c398-ca3f-4378-9eb9-93ce9bac8db5\") " pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"
Sep 10 19:59:42.600567 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.600375    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bb2c398-ca3f-4378-9eb9-93ce9bac8db5-tmp\") pod \"kube-auth-proxy-576c9858ff-hknzf\" (UID: \"7bb2c398-ca3f-4378-9eb9-93ce9bac8db5\") " pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"
Sep 10 19:59:42.600567 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.600408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrhxf\" (UniqueName: \"kubernetes.io/projected/7bb2c398-ca3f-4378-9eb9-93ce9bac8db5-kube-api-access-qrhxf\") pod \"kube-auth-proxy-576c9858ff-hknzf\" (UID: \"7bb2c398-ca3f-4378-9eb9-93ce9bac8db5\") " pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"
Sep 10 19:59:42.600567 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:59:42.600474    2561 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 19:59:42.600683 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:59:42.600582    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb2c398-ca3f-4378-9eb9-93ce9bac8db5-tls-certs podName:7bb2c398-ca3f-4378-9eb9-93ce9bac8db5 nodeName:}" failed. No retries permitted until 2026-09-10 19:59:43.100560222 +0000 UTC m=+324.151088194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7bb2c398-ca3f-4378-9eb9-93ce9bac8db5-tls-certs") pod "kube-auth-proxy-576c9858ff-hknzf" (UID: "7bb2c398-ca3f-4378-9eb9-93ce9bac8db5") : secret "kube-auth-proxy-tls" not found
Sep 10 19:59:42.602789 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.602769    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bb2c398-ca3f-4378-9eb9-93ce9bac8db5-tmp\") pod \"kube-auth-proxy-576c9858ff-hknzf\" (UID: \"7bb2c398-ca3f-4378-9eb9-93ce9bac8db5\") " pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"
Sep 10 19:59:42.610718 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.610696    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrhxf\" (UniqueName: \"kubernetes.io/projected/7bb2c398-ca3f-4378-9eb9-93ce9bac8db5-kube-api-access-qrhxf\") pod \"kube-auth-proxy-576c9858ff-hknzf\" (UID: \"7bb2c398-ca3f-4378-9eb9-93ce9bac8db5\") " pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"
Sep 10 19:59:42.832265 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:42.832230    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:43.105355 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:43.105266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb2c398-ca3f-4378-9eb9-93ce9bac8db5-tls-certs\") pod \"kube-auth-proxy-576c9858ff-hknzf\" (UID: \"7bb2c398-ca3f-4378-9eb9-93ce9bac8db5\") " pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"
Sep 10 19:59:43.107941 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:43.107920    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb2c398-ca3f-4378-9eb9-93ce9bac8db5-tls-certs\") pod \"kube-auth-proxy-576c9858ff-hknzf\" (UID: \"7bb2c398-ca3f-4378-9eb9-93ce9bac8db5\") " pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"
Sep 10 19:59:43.323111 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:43.323068    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"
Sep 10 19:59:43.599721 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:43.599694    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-576c9858ff-hknzf"]
Sep 10 19:59:43.601809 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:59:43.601787    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb2c398_ca3f_4378_9eb9_93ce9bac8db5.slice/crio-84d3ccf0e3ffa97c02eeb2954eae4cdf0ed4b0a9f66e29d51a82135afbc1dc7e WatchSource:0}: Error finding container 84d3ccf0e3ffa97c02eeb2954eae4cdf0ed4b0a9f66e29d51a82135afbc1dc7e: Status 404 returned error can't find the container with id 84d3ccf0e3ffa97c02eeb2954eae4cdf0ed4b0a9f66e29d51a82135afbc1dc7e
Sep 10 19:59:43.898413 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:43.898378    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf" event={"ID":"7bb2c398-ca3f-4378-9eb9-93ce9bac8db5","Type":"ContainerStarted","Data":"84d3ccf0e3ffa97c02eeb2954eae4cdf0ed4b0a9f66e29d51a82135afbc1dc7e"}
Sep 10 19:59:43.951126 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:43.951095    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:44.026227 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:44.026172    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:45.289112 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:45.289049    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:46.918846 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:46.918816    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf" event={"ID":"7bb2c398-ca3f-4378-9eb9-93ce9bac8db5","Type":"ContainerStarted","Data":"1beb4f4f5ff626dc718210b1caf8f0fc6ebd7f2f6a2d48f03d7d49e82f383219"}
Sep 10 19:59:46.950064 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:46.950008    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-576c9858ff-hknzf" podStartSLOduration=2.229334097 podStartE2EDuration="4.949990192s" totalImagesPullingTime="2.720656095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:59:46.948945363 +0000 UTC m=+327.999473369" watchObservedRunningTime="2026-09-10 19:59:46.949990192 +0000 UTC m=+328.000518181"
Sep 10 19:59:47.641860 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:47.641828    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f8hnl"]
Sep 10 19:59:47.645093 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:47.645077    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-f8hnl"
Sep 10 19:59:47.647848 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:47.647825    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4s5t5\""
Sep 10 19:59:47.647978 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:47.647961    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 10 19:59:47.655257 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:47.655236    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f8hnl"]
Sep 10 19:59:47.686107 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:47.686077    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:47.743037 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:47.743005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjbhl\" (UniqueName: \"kubernetes.io/projected/84d2936e-2f1f-4334-a600-4dc6e33172d5-kube-api-access-rjbhl\") pod \"odh-model-controller-7998c67dc4-f8hnl\" (UID: \"84d2936e-2f1f-4334-a600-4dc6e33172d5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8hnl"
Sep 10 19:59:47.743217 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:47.743058    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84d2936e-2f1f-4334-a600-4dc6e33172d5-cert\") pod \"odh-model-controller-7998c67dc4-f8hnl\" (UID: \"84d2936e-2f1f-4334-a600-4dc6e33172d5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8hnl"
Sep 10 19:59:47.785827 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:47.785791    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:47.844585 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:47.844522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84d2936e-2f1f-4334-a600-4dc6e33172d5-cert\") pod \"odh-model-controller-7998c67dc4-f8hnl\" (UID: \"84d2936e-2f1f-4334-a600-4dc6e33172d5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8hnl"
Sep 10 19:59:47.844753 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:47.844627    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjbhl\" (UniqueName: \"kubernetes.io/projected/84d2936e-2f1f-4334-a600-4dc6e33172d5-kube-api-access-rjbhl\") pod \"odh-model-controller-7998c67dc4-f8hnl\" (UID: \"84d2936e-2f1f-4334-a600-4dc6e33172d5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8hnl"
Sep 10 19:59:47.844753 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:59:47.844681    2561 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 10 19:59:47.844824 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:59:47.844755    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d2936e-2f1f-4334-a600-4dc6e33172d5-cert podName:84d2936e-2f1f-4334-a600-4dc6e33172d5 nodeName:}" failed. No retries permitted until 2026-09-10 19:59:48.344737119 +0000 UTC m=+329.395265105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/84d2936e-2f1f-4334-a600-4dc6e33172d5-cert") pod "odh-model-controller-7998c67dc4-f8hnl" (UID: "84d2936e-2f1f-4334-a600-4dc6e33172d5") : secret "odh-model-controller-webhook-cert" not found
Sep 10 19:59:47.856519 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:47.856485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjbhl\" (UniqueName: \"kubernetes.io/projected/84d2936e-2f1f-4334-a600-4dc6e33172d5-kube-api-access-rjbhl\") pod \"odh-model-controller-7998c67dc4-f8hnl\" (UID: \"84d2936e-2f1f-4334-a600-4dc6e33172d5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8hnl"
Sep 10 19:59:47.885435 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:47.885398    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:47.985878 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:47.985800    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:48.334607 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.334570    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:48.348698 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.348667    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84d2936e-2f1f-4334-a600-4dc6e33172d5-cert\") pod \"odh-model-controller-7998c67dc4-f8hnl\" (UID: \"84d2936e-2f1f-4334-a600-4dc6e33172d5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8hnl"
Sep 10 19:59:48.351256 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.351235    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84d2936e-2f1f-4334-a600-4dc6e33172d5-cert\") pod \"odh-model-controller-7998c67dc4-f8hnl\" (UID: \"84d2936e-2f1f-4334-a600-4dc6e33172d5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8hnl"
Sep 10 19:59:48.373129 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.373100    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-57hqs"]
Sep 10 19:59:48.376550 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.376517    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-57hqs"
Sep 10 19:59:48.379849 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.379833    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-gdvwp\""
Sep 10 19:59:48.380103 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.380087    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 19:59:48.395956 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.395935    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-57hqs"]
Sep 10 19:59:48.449420 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.449386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/810e302d-18ea-4e8d-b0c3-68c974aa7cc0-tls-certs\") pod \"model-serving-api-54d89bd5fc-57hqs\" (UID: \"810e302d-18ea-4e8d-b0c3-68c974aa7cc0\") " pod="opendatahub/model-serving-api-54d89bd5fc-57hqs"
Sep 10 19:59:48.449614 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.449492    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slr94\" (UniqueName: \"kubernetes.io/projected/810e302d-18ea-4e8d-b0c3-68c974aa7cc0-kube-api-access-slr94\") pod \"model-serving-api-54d89bd5fc-57hqs\" (UID: \"810e302d-18ea-4e8d-b0c3-68c974aa7cc0\") " pod="opendatahub/model-serving-api-54d89bd5fc-57hqs"
Sep 10 19:59:48.550949 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.550915    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slr94\" (UniqueName: \"kubernetes.io/projected/810e302d-18ea-4e8d-b0c3-68c974aa7cc0-kube-api-access-slr94\") pod \"model-serving-api-54d89bd5fc-57hqs\" (UID: \"810e302d-18ea-4e8d-b0c3-68c974aa7cc0\") " pod="opendatahub/model-serving-api-54d89bd5fc-57hqs"
Sep 10 19:59:48.550949 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.550955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/810e302d-18ea-4e8d-b0c3-68c974aa7cc0-tls-certs\") pod \"model-serving-api-54d89bd5fc-57hqs\" (UID: \"810e302d-18ea-4e8d-b0c3-68c974aa7cc0\") " pod="opendatahub/model-serving-api-54d89bd5fc-57hqs"
Sep 10 19:59:48.551146 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:59:48.551061    2561 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 10 19:59:48.551146 ip-10-0-140-111 kubenswrapper[2561]: E0910 19:59:48.551114    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810e302d-18ea-4e8d-b0c3-68c974aa7cc0-tls-certs podName:810e302d-18ea-4e8d-b0c3-68c974aa7cc0 nodeName:}" failed. No retries permitted until 2026-09-10 19:59:49.051097755 +0000 UTC m=+330.101625721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/810e302d-18ea-4e8d-b0c3-68c974aa7cc0-tls-certs") pod "model-serving-api-54d89bd5fc-57hqs" (UID: "810e302d-18ea-4e8d-b0c3-68c974aa7cc0") : secret "model-serving-api-tls" not found
Sep 10 19:59:48.555997 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.555979    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-f8hnl"
Sep 10 19:59:48.561341 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.561318    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slr94\" (UniqueName: \"kubernetes.io/projected/810e302d-18ea-4e8d-b0c3-68c974aa7cc0-kube-api-access-slr94\") pod \"model-serving-api-54d89bd5fc-57hqs\" (UID: \"810e302d-18ea-4e8d-b0c3-68c974aa7cc0\") " pod="opendatahub/model-serving-api-54d89bd5fc-57hqs"
Sep 10 19:59:48.657800 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.657764    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:48.828417 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.828393    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f8hnl"]
Sep 10 19:59:48.830365 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:59:48.830342    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d2936e_2f1f_4334_a600_4dc6e33172d5.slice/crio-8c1f4c65e13d276c1c663090ce976f756c21a9cb1d14657a2f1a623fdacaaed3 WatchSource:0}: Error finding container 8c1f4c65e13d276c1c663090ce976f756c21a9cb1d14657a2f1a623fdacaaed3: Status 404 returned error can't find the container with id 8c1f4c65e13d276c1c663090ce976f756c21a9cb1d14657a2f1a623fdacaaed3
Sep 10 19:59:48.929358 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:48.929276    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-f8hnl" event={"ID":"84d2936e-2f1f-4334-a600-4dc6e33172d5","Type":"ContainerStarted","Data":"8c1f4c65e13d276c1c663090ce976f756c21a9cb1d14657a2f1a623fdacaaed3"}
Sep 10 19:59:49.055021 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:49.054981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/810e302d-18ea-4e8d-b0c3-68c974aa7cc0-tls-certs\") pod \"model-serving-api-54d89bd5fc-57hqs\" (UID: \"810e302d-18ea-4e8d-b0c3-68c974aa7cc0\") " pod="opendatahub/model-serving-api-54d89bd5fc-57hqs"
Sep 10 19:59:49.057631 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:49.057604    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/810e302d-18ea-4e8d-b0c3-68c974aa7cc0-tls-certs\") pod \"model-serving-api-54d89bd5fc-57hqs\" (UID: \"810e302d-18ea-4e8d-b0c3-68c974aa7cc0\") " pod="opendatahub/model-serving-api-54d89bd5fc-57hqs"
Sep 10 19:59:49.079160 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:49.079126    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:49.226961 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:49.226883    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:49.286943 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:49.286907    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-57hqs"
Sep 10 19:59:49.568189 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:49.568160    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-57hqs"]
Sep 10 19:59:49.569234 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:59:49.569208    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810e302d_18ea_4e8d_b0c3_68c974aa7cc0.slice/crio-afca1e3e712bab802128a73cbbc4c76a8fb077a6651c37223f9c2cd47578d125 WatchSource:0}: Error finding container afca1e3e712bab802128a73cbbc4c76a8fb077a6651c37223f9c2cd47578d125: Status 404 returned error can't find the container with id afca1e3e712bab802128a73cbbc4c76a8fb077a6651c37223f9c2cd47578d125
Sep 10 19:59:49.937139 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:49.937042    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-57hqs" event={"ID":"810e302d-18ea-4e8d-b0c3-68c974aa7cc0","Type":"ContainerStarted","Data":"afca1e3e712bab802128a73cbbc4c76a8fb077a6651c37223f9c2cd47578d125"}
Sep 10 19:59:50.051628 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:50.051590    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:50.338960 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:50.338928    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:50.439658 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:50.439596    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:52.956236 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:52.956199    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-f8hnl" event={"ID":"84d2936e-2f1f-4334-a600-4dc6e33172d5","Type":"ContainerStarted","Data":"b1ef8259b540dacddfd33efaa642e14a51acc22ca85e06b46b4d495be73addd5"}
Sep 10 19:59:52.956698 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:52.956332    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-f8hnl"
Sep 10 19:59:52.957761 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:52.957740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-57hqs" event={"ID":"810e302d-18ea-4e8d-b0c3-68c974aa7cc0","Type":"ContainerStarted","Data":"11548d35ffe6449e0b3ec7320958cc8883853bd428a568fa72e4807971b1c6bc"}
Sep 10 19:59:52.957860 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:52.957788    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-57hqs"
Sep 10 19:59:52.981945 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:52.981899    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-f8hnl" podStartSLOduration=2.593057124 podStartE2EDuration="5.981884055s" totalImagesPullingTime="3.388826931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:59:52.979478249 +0000 UTC m=+334.030006237" watchObservedRunningTime="2026-09-10 19:59:52.981884055 +0000 UTC m=+334.032412088"
Sep 10 19:59:53.012340 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:53.012284    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-57hqs" podStartSLOduration=2.363546432 podStartE2EDuration="5.012267845s" totalImagesPullingTime="2.648721413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:59:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:59:53.010969706 +0000 UTC m=+334.061497699" watchObservedRunningTime="2026-09-10 19:59:53.012267845 +0000 UTC m=+334.062795834"
Sep 10 19:59:54.844624 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:54.844586    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"]
Sep 10 19:59:54.847093 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:54.847070    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:54.850046 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:54.850022    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 19:59:54.851009 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:54.850988    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 19:59:54.851114 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:54.851020    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 19:59:54.851114 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:54.851038    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ss24s\""
Sep 10 19:59:54.861191 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:54.861168    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"]
Sep 10 19:59:54.905054 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:54.905018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24ftk\" (UniqueName: \"kubernetes.io/projected/738906e4-a1a1-4db4-8de1-2ce79e3a548f-kube-api-access-24ftk\") pod \"lws-controller-manager-6fd976b7bd-qkcsb\" (UID: \"738906e4-a1a1-4db4-8de1-2ce79e3a548f\") " pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:54.905054 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:54.905053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/738906e4-a1a1-4db4-8de1-2ce79e3a548f-cert\") pod \"lws-controller-manager-6fd976b7bd-qkcsb\" (UID: \"738906e4-a1a1-4db4-8de1-2ce79e3a548f\") " pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:54.905260 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:54.905094    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/738906e4-a1a1-4db4-8de1-2ce79e3a548f-metrics-cert\") pod \"lws-controller-manager-6fd976b7bd-qkcsb\" (UID: \"738906e4-a1a1-4db4-8de1-2ce79e3a548f\") " pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:54.905260 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:54.905184    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/738906e4-a1a1-4db4-8de1-2ce79e3a548f-manager-config\") pod \"lws-controller-manager-6fd976b7bd-qkcsb\" (UID: \"738906e4-a1a1-4db4-8de1-2ce79e3a548f\") " pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:55.006159 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:55.006128    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24ftk\" (UniqueName: \"kubernetes.io/projected/738906e4-a1a1-4db4-8de1-2ce79e3a548f-kube-api-access-24ftk\") pod \"lws-controller-manager-6fd976b7bd-qkcsb\" (UID: \"738906e4-a1a1-4db4-8de1-2ce79e3a548f\") " pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:55.006159 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:55.006163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/738906e4-a1a1-4db4-8de1-2ce79e3a548f-cert\") pod \"lws-controller-manager-6fd976b7bd-qkcsb\" (UID: \"738906e4-a1a1-4db4-8de1-2ce79e3a548f\") " pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:55.006370 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:55.006209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/738906e4-a1a1-4db4-8de1-2ce79e3a548f-metrics-cert\") pod \"lws-controller-manager-6fd976b7bd-qkcsb\" (UID: \"738906e4-a1a1-4db4-8de1-2ce79e3a548f\") " pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:55.006370 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:55.006263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/738906e4-a1a1-4db4-8de1-2ce79e3a548f-manager-config\") pod \"lws-controller-manager-6fd976b7bd-qkcsb\" (UID: \"738906e4-a1a1-4db4-8de1-2ce79e3a548f\") " pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:55.006911 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:55.006887    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/738906e4-a1a1-4db4-8de1-2ce79e3a548f-manager-config\") pod \"lws-controller-manager-6fd976b7bd-qkcsb\" (UID: \"738906e4-a1a1-4db4-8de1-2ce79e3a548f\") " pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:55.008897 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:55.008870    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/738906e4-a1a1-4db4-8de1-2ce79e3a548f-metrics-cert\") pod \"lws-controller-manager-6fd976b7bd-qkcsb\" (UID: \"738906e4-a1a1-4db4-8de1-2ce79e3a548f\") " pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:55.009002 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:55.008903    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/738906e4-a1a1-4db4-8de1-2ce79e3a548f-cert\") pod \"lws-controller-manager-6fd976b7bd-qkcsb\" (UID: \"738906e4-a1a1-4db4-8de1-2ce79e3a548f\") " pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:55.020019 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:55.019998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24ftk\" (UniqueName: \"kubernetes.io/projected/738906e4-a1a1-4db4-8de1-2ce79e3a548f-kube-api-access-24ftk\") pod \"lws-controller-manager-6fd976b7bd-qkcsb\" (UID: \"738906e4-a1a1-4db4-8de1-2ce79e3a548f\") " pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:55.157024 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:55.156943    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:55.439800 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:55.439757    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"]
Sep 10 19:59:55.442157 ip-10-0-140-111 kubenswrapper[2561]: W0910 19:59:55.442125    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738906e4_a1a1_4db4_8de1_2ce79e3a548f.slice/crio-27c0e381b18e60a443fc34c35d31551f1aea6430c5058b0f045418d860eedd71 WatchSource:0}: Error finding container 27c0e381b18e60a443fc34c35d31551f1aea6430c5058b0f045418d860eedd71: Status 404 returned error can't find the container with id 27c0e381b18e60a443fc34c35d31551f1aea6430c5058b0f045418d860eedd71
Sep 10 19:59:55.973079 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:55.973045    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb" event={"ID":"738906e4-a1a1-4db4-8de1-2ce79e3a548f","Type":"ContainerStarted","Data":"27c0e381b18e60a443fc34c35d31551f1aea6430c5058b0f045418d860eedd71"}
Sep 10 19:59:57.985316 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:57.985274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb" event={"ID":"738906e4-a1a1-4db4-8de1-2ce79e3a548f","Type":"ContainerStarted","Data":"d441cd78aa638bc647bdbdb9acc8551d774dd078e9c954490b702a78134cb7c6"}
Sep 10 19:59:57.985820 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:57.985404    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 19:59:58.257373 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:58.257288    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:58.475509 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:58.475479    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:58.749686 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:58.749644    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:58.993185 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:58.993137    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:59.756204 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:59.756171    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 19:59:59.960155 ip-10-0-140-111 kubenswrapper[2561]: I0910 19:59:59.960124    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:00:03.965235 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:03.965202    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-f8hnl"
Sep 10 20:00:03.967343 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:03.967322    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-57hqs"
Sep 10 20:00:03.986293 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:03.986246    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb" podStartSLOduration=8.50084568 podStartE2EDuration="9.986230452s" totalImagesPullingTime="1.485384772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:59:58.033429899 +0000 UTC m=+339.083957888" watchObservedRunningTime="2026-09-10 20:00:03.986230452 +0000 UTC m=+345.036758474"
Sep 10 20:00:08.992452 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:08.992419    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6fd976b7bd-qkcsb"
Sep 10 20:00:48.858698 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:48.858659    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:00:48.994389 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:48.994359    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:00:54.032455 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.032419    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"]
Sep 10 20:00:54.036732 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.036713    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"
Sep 10 20:00:54.039343 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.039320    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 20:00:54.039478 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.039343    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 20:00:54.039617 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.039571    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-knjsq\""
Sep 10 20:00:54.050102 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.050069    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"]
Sep 10 20:00:54.108185 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.108148    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldvft\" (UniqueName: \"kubernetes.io/projected/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-kube-api-access-ldvft\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx\" (UID: \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"
Sep 10 20:00:54.108367 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.108208    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx\" (UID: \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"
Sep 10 20:00:54.108367 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.108287    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx\" (UID: \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"
Sep 10 20:00:54.128917 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.128882    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"]
Sep 10 20:00:54.131875 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.131858    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"
Sep 10 20:00:54.151584 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.151543    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"]
Sep 10 20:00:54.209126 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.209085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldvft\" (UniqueName: \"kubernetes.io/projected/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-kube-api-access-ldvft\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx\" (UID: \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"
Sep 10 20:00:54.209331 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.209140    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f\" (UID: \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"
Sep 10 20:00:54.209331 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.209187    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx\" (UID: \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"
Sep 10 20:00:54.209331 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.209203    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f\" (UID: \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"
Sep 10 20:00:54.209331 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.209231    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl4mf\" (UniqueName: \"kubernetes.io/projected/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-kube-api-access-jl4mf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f\" (UID: \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"
Sep 10 20:00:54.209331 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.209316    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx\" (UID: \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"
Sep 10 20:00:54.209641 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.209624    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx\" (UID: \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"
Sep 10 20:00:54.209679 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.209659    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx\" (UID: \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"
Sep 10 20:00:54.218465 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.218437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldvft\" (UniqueName: \"kubernetes.io/projected/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-kube-api-access-ldvft\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx\" (UID: \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"
Sep 10 20:00:54.234913 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.234881    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"]
Sep 10 20:00:54.237402 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.237386    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"
Sep 10 20:00:54.248570 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.248543    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"]
Sep 10 20:00:54.309941 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.309851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdbf8ab5-e278-412d-be46-d33f4a749b72-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd\" (UID: \"cdbf8ab5-e278-412d-be46-d33f4a749b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"
Sep 10 20:00:54.309941 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.309929    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhngq\" (UniqueName: \"kubernetes.io/projected/cdbf8ab5-e278-412d-be46-d33f4a749b72-kube-api-access-lhngq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd\" (UID: \"cdbf8ab5-e278-412d-be46-d33f4a749b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"
Sep 10 20:00:54.310163 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.309958    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f\" (UID: \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"
Sep 10 20:00:54.310163 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.309979    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdbf8ab5-e278-412d-be46-d33f4a749b72-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd\" (UID: \"cdbf8ab5-e278-412d-be46-d33f4a749b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"
Sep 10 20:00:54.310163 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.310057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f\" (UID: \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"
Sep 10 20:00:54.310163 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.310112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl4mf\" (UniqueName: \"kubernetes.io/projected/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-kube-api-access-jl4mf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f\" (UID: \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"
Sep 10 20:00:54.310379 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.310359    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f\" (UID: \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"
Sep 10 20:00:54.310472 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.310454    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f\" (UID: \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"
Sep 10 20:00:54.322080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.322055    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl4mf\" (UniqueName: \"kubernetes.io/projected/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-kube-api-access-jl4mf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f\" (UID: \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"
Sep 10 20:00:54.336490 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.336458    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"]
Sep 10 20:00:54.339056 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.339041    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"
Sep 10 20:00:54.348473 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.348449    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"
Sep 10 20:00:54.354068 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.354040    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"]
Sep 10 20:00:54.410729 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.410697    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/542b98ae-7412-4b81-a759-77f2355c2055-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx\" (UID: \"542b98ae-7412-4b81-a759-77f2355c2055\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"
Sep 10 20:00:54.410892 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.410791    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdbf8ab5-e278-412d-be46-d33f4a749b72-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd\" (UID: \"cdbf8ab5-e278-412d-be46-d33f4a749b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"
Sep 10 20:00:54.410892 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.410848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/542b98ae-7412-4b81-a759-77f2355c2055-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx\" (UID: \"542b98ae-7412-4b81-a759-77f2355c2055\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"
Sep 10 20:00:54.411024 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.410889    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhngq\" (UniqueName: \"kubernetes.io/projected/cdbf8ab5-e278-412d-be46-d33f4a749b72-kube-api-access-lhngq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd\" (UID: \"cdbf8ab5-e278-412d-be46-d33f4a749b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"
Sep 10 20:00:54.411024 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.410925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdbf8ab5-e278-412d-be46-d33f4a749b72-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd\" (UID: \"cdbf8ab5-e278-412d-be46-d33f4a749b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"
Sep 10 20:00:54.411024 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.410953    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjwvl\" (UniqueName: \"kubernetes.io/projected/542b98ae-7412-4b81-a759-77f2355c2055-kube-api-access-rjwvl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx\" (UID: \"542b98ae-7412-4b81-a759-77f2355c2055\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"
Sep 10 20:00:54.411590 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.411367    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdbf8ab5-e278-412d-be46-d33f4a749b72-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd\" (UID: \"cdbf8ab5-e278-412d-be46-d33f4a749b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"
Sep 10 20:00:54.411590 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.411456    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdbf8ab5-e278-412d-be46-d33f4a749b72-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd\" (UID: \"cdbf8ab5-e278-412d-be46-d33f4a749b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"
Sep 10 20:00:54.420444 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.420401    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhngq\" (UniqueName: \"kubernetes.io/projected/cdbf8ab5-e278-412d-be46-d33f4a749b72-kube-api-access-lhngq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd\" (UID: \"cdbf8ab5-e278-412d-be46-d33f4a749b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"
Sep 10 20:00:54.441273 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.441246    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"
Sep 10 20:00:54.512097 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.512065    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/542b98ae-7412-4b81-a759-77f2355c2055-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx\" (UID: \"542b98ae-7412-4b81-a759-77f2355c2055\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"
Sep 10 20:00:54.512271 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.512158    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/542b98ae-7412-4b81-a759-77f2355c2055-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx\" (UID: \"542b98ae-7412-4b81-a759-77f2355c2055\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"
Sep 10 20:00:54.512271 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.512207    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjwvl\" (UniqueName: \"kubernetes.io/projected/542b98ae-7412-4b81-a759-77f2355c2055-kube-api-access-rjwvl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx\" (UID: \"542b98ae-7412-4b81-a759-77f2355c2055\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"
Sep 10 20:00:54.512561 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.512539    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/542b98ae-7412-4b81-a759-77f2355c2055-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx\" (UID: \"542b98ae-7412-4b81-a759-77f2355c2055\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"
Sep 10 20:00:54.512643 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.512595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/542b98ae-7412-4b81-a759-77f2355c2055-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx\" (UID: \"542b98ae-7412-4b81-a759-77f2355c2055\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"
Sep 10 20:00:54.522193 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.522166    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjwvl\" (UniqueName: \"kubernetes.io/projected/542b98ae-7412-4b81-a759-77f2355c2055-kube-api-access-rjwvl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx\" (UID: \"542b98ae-7412-4b81-a759-77f2355c2055\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"
Sep 10 20:00:54.547069 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.547036    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"
Sep 10 20:00:54.630935 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.630907    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"]
Sep 10 20:00:54.632522 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:00:54.632492    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1557366b_fd3a_4a4c_bceb_cc80f69e7d58.slice/crio-7ceaa0cb8eede6a10cab969adef341d61da1ddb543866288145e39306985622a WatchSource:0}: Error finding container 7ceaa0cb8eede6a10cab969adef341d61da1ddb543866288145e39306985622a: Status 404 returned error can't find the container with id 7ceaa0cb8eede6a10cab969adef341d61da1ddb543866288145e39306985622a
Sep 10 20:00:54.650294 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.648284    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"
Sep 10 20:00:54.717715 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.717687    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"]
Sep 10 20:00:54.719980 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:00:54.719920    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb55d0cd7_7c64_4d67_a7dc_032c1ee06728.slice/crio-e2a8ac6fb16765d79fa6a4f837a6cd1ad34a66bf880b0095d73632756cfa2ffc WatchSource:0}: Error finding container e2a8ac6fb16765d79fa6a4f837a6cd1ad34a66bf880b0095d73632756cfa2ffc: Status 404 returned error can't find the container with id e2a8ac6fb16765d79fa6a4f837a6cd1ad34a66bf880b0095d73632756cfa2ffc
Sep 10 20:00:54.820076 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.820048    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"]
Sep 10 20:00:54.822117 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:00:54.822091    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbf8ab5_e278_412d_be46_d33f4a749b72.slice/crio-a07c30bb74fe939a0b426b2b10d2c8881247fccfa49b33820c1a5b31ffec112f WatchSource:0}: Error finding container a07c30bb74fe939a0b426b2b10d2c8881247fccfa49b33820c1a5b31ffec112f: Status 404 returned error can't find the container with id a07c30bb74fe939a0b426b2b10d2c8881247fccfa49b33820c1a5b31ffec112f
Sep 10 20:00:54.928336 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:54.928308    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"]
Sep 10 20:00:54.971473 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:00:54.971429    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542b98ae_7412_4b81_a759_77f2355c2055.slice/crio-19226855efbdad69721f2112f4531f0e59100637a8c6703516be2d797695abb3 WatchSource:0}: Error finding container 19226855efbdad69721f2112f4531f0e59100637a8c6703516be2d797695abb3: Status 404 returned error can't find the container with id 19226855efbdad69721f2112f4531f0e59100637a8c6703516be2d797695abb3
Sep 10 20:00:55.272776 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:55.272735    2561 generic.go:356] "Generic (PLEG): container finished" podID="b55d0cd7-7c64-4d67-a7dc-032c1ee06728" containerID="525fabd777edb6ada69981851f453411c6ae0afc409d8a4c67b94758cb1b4a15" exitCode=0
Sep 10 20:00:55.273219 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:55.272828    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f" event={"ID":"b55d0cd7-7c64-4d67-a7dc-032c1ee06728","Type":"ContainerDied","Data":"525fabd777edb6ada69981851f453411c6ae0afc409d8a4c67b94758cb1b4a15"}
Sep 10 20:00:55.273219 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:55.272884    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f" event={"ID":"b55d0cd7-7c64-4d67-a7dc-032c1ee06728","Type":"ContainerStarted","Data":"e2a8ac6fb16765d79fa6a4f837a6cd1ad34a66bf880b0095d73632756cfa2ffc"}
Sep 10 20:00:55.274604 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:55.274584    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdbf8ab5-e278-412d-be46-d33f4a749b72" containerID="afa11e37738a45a7636344536cbe206eab8272039487ac6aa3b64b3a672a9ad9" exitCode=0
Sep 10 20:00:55.274684 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:55.274638    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd" event={"ID":"cdbf8ab5-e278-412d-be46-d33f4a749b72","Type":"ContainerDied","Data":"afa11e37738a45a7636344536cbe206eab8272039487ac6aa3b64b3a672a9ad9"}
Sep 10 20:00:55.274737 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:55.274683    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd" event={"ID":"cdbf8ab5-e278-412d-be46-d33f4a749b72","Type":"ContainerStarted","Data":"a07c30bb74fe939a0b426b2b10d2c8881247fccfa49b33820c1a5b31ffec112f"}
Sep 10 20:00:55.276220 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:55.276201    2561 generic.go:356] "Generic (PLEG): container finished" podID="1557366b-fd3a-4a4c-bceb-cc80f69e7d58" containerID="e717d3a0c3dc55291c34b2e390d8898de2f071793a30ead1e391870b5a35431a" exitCode=0
Sep 10 20:00:55.276293 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:55.276274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx" event={"ID":"1557366b-fd3a-4a4c-bceb-cc80f69e7d58","Type":"ContainerDied","Data":"e717d3a0c3dc55291c34b2e390d8898de2f071793a30ead1e391870b5a35431a"}
Sep 10 20:00:55.276340 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:55.276315    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx" event={"ID":"1557366b-fd3a-4a4c-bceb-cc80f69e7d58","Type":"ContainerStarted","Data":"7ceaa0cb8eede6a10cab969adef341d61da1ddb543866288145e39306985622a"}
Sep 10 20:00:55.278054 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:55.278033    2561 generic.go:356] "Generic (PLEG): container finished" podID="542b98ae-7412-4b81-a759-77f2355c2055" containerID="13ee5302c73ef34f7a257f23fbedf92cc599dec436385c0aafc027e80f6831ae" exitCode=0
Sep 10 20:00:55.278135 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:55.278071    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx" event={"ID":"542b98ae-7412-4b81-a759-77f2355c2055","Type":"ContainerDied","Data":"13ee5302c73ef34f7a257f23fbedf92cc599dec436385c0aafc027e80f6831ae"}
Sep 10 20:00:55.278135 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:55.278115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx" event={"ID":"542b98ae-7412-4b81-a759-77f2355c2055","Type":"ContainerStarted","Data":"19226855efbdad69721f2112f4531f0e59100637a8c6703516be2d797695abb3"}
Sep 10 20:00:56.289949 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:56.289910    2561 generic.go:356] "Generic (PLEG): container finished" podID="542b98ae-7412-4b81-a759-77f2355c2055" containerID="e9d128278cbec9a9093c859c7fb0476838188202e251d64171a59f08a36ec314" exitCode=0
Sep 10 20:00:56.290321 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:56.289962    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx" event={"ID":"542b98ae-7412-4b81-a759-77f2355c2055","Type":"ContainerDied","Data":"e9d128278cbec9a9093c859c7fb0476838188202e251d64171a59f08a36ec314"}
Sep 10 20:00:57.296949 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:57.296911    2561 generic.go:356] "Generic (PLEG): container finished" podID="b55d0cd7-7c64-4d67-a7dc-032c1ee06728" containerID="06e807da1d33e8bdcf331e22fe81c7d88b8473f7eb6e767f84d53b91fc447187" exitCode=0
Sep 10 20:00:57.297376 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:57.296961    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f" event={"ID":"b55d0cd7-7c64-4d67-a7dc-032c1ee06728","Type":"ContainerDied","Data":"06e807da1d33e8bdcf331e22fe81c7d88b8473f7eb6e767f84d53b91fc447187"}
Sep 10 20:00:57.298981 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:57.298961    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdbf8ab5-e278-412d-be46-d33f4a749b72" containerID="ea4aad6a25ba48ac6db853599eb4a542c5dbbdf65028da1ab4425486104a1df5" exitCode=0
Sep 10 20:00:57.299090 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:57.299027    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd" event={"ID":"cdbf8ab5-e278-412d-be46-d33f4a749b72","Type":"ContainerDied","Data":"ea4aad6a25ba48ac6db853599eb4a542c5dbbdf65028da1ab4425486104a1df5"}
Sep 10 20:00:57.300893 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:57.300872    2561 generic.go:356] "Generic (PLEG): container finished" podID="1557366b-fd3a-4a4c-bceb-cc80f69e7d58" containerID="7b803d09e157c74a27d2678c172b4e95885ee4f5e3f38e9f7241c44ffa71b90c" exitCode=0
Sep 10 20:00:57.300985 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:57.300951    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx" event={"ID":"1557366b-fd3a-4a4c-bceb-cc80f69e7d58","Type":"ContainerDied","Data":"7b803d09e157c74a27d2678c172b4e95885ee4f5e3f38e9f7241c44ffa71b90c"}
Sep 10 20:00:57.303429 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:57.303407    2561 generic.go:356] "Generic (PLEG): container finished" podID="542b98ae-7412-4b81-a759-77f2355c2055" containerID="530409acaf04ea81a31f0b6b1d2b2306c65f6bf8aff1e67d9c7badb6e11f0620" exitCode=0
Sep 10 20:00:57.303549 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:57.303447    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx" event={"ID":"542b98ae-7412-4b81-a759-77f2355c2055","Type":"ContainerDied","Data":"530409acaf04ea81a31f0b6b1d2b2306c65f6bf8aff1e67d9c7badb6e11f0620"}
Sep 10 20:00:58.311242 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.311209    2561 generic.go:356] "Generic (PLEG): container finished" podID="b55d0cd7-7c64-4d67-a7dc-032c1ee06728" containerID="2138aa8db42c763705a863dcf6f2f6ea3347862a202f50e7644145552c5e59ed" exitCode=0
Sep 10 20:00:58.311751 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.311292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f" event={"ID":"b55d0cd7-7c64-4d67-a7dc-032c1ee06728","Type":"ContainerDied","Data":"2138aa8db42c763705a863dcf6f2f6ea3347862a202f50e7644145552c5e59ed"}
Sep 10 20:00:58.313466 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.313445    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdbf8ab5-e278-412d-be46-d33f4a749b72" containerID="6360751c6507a81fac042f3d86c447bdce70ff0a4ea398b8f0a6433d641e9758" exitCode=0
Sep 10 20:00:58.313577 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.313509    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd" event={"ID":"cdbf8ab5-e278-412d-be46-d33f4a749b72","Type":"ContainerDied","Data":"6360751c6507a81fac042f3d86c447bdce70ff0a4ea398b8f0a6433d641e9758"}
Sep 10 20:00:58.315758 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.315727    2561 generic.go:356] "Generic (PLEG): container finished" podID="1557366b-fd3a-4a4c-bceb-cc80f69e7d58" containerID="e2882a57f627af6a11b27eb35395d834f57097fc9a4e900a42944650cb9846af" exitCode=0
Sep 10 20:00:58.315861 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.315755    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx" event={"ID":"1557366b-fd3a-4a4c-bceb-cc80f69e7d58","Type":"ContainerDied","Data":"e2882a57f627af6a11b27eb35395d834f57097fc9a4e900a42944650cb9846af"}
Sep 10 20:00:58.454032 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.454009    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"
Sep 10 20:00:58.549173 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.549141    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/542b98ae-7412-4b81-a759-77f2355c2055-util\") pod \"542b98ae-7412-4b81-a759-77f2355c2055\" (UID: \"542b98ae-7412-4b81-a759-77f2355c2055\") "
Sep 10 20:00:58.549336 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.549201    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjwvl\" (UniqueName: \"kubernetes.io/projected/542b98ae-7412-4b81-a759-77f2355c2055-kube-api-access-rjwvl\") pod \"542b98ae-7412-4b81-a759-77f2355c2055\" (UID: \"542b98ae-7412-4b81-a759-77f2355c2055\") "
Sep 10 20:00:58.549336 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.549228    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/542b98ae-7412-4b81-a759-77f2355c2055-bundle\") pod \"542b98ae-7412-4b81-a759-77f2355c2055\" (UID: \"542b98ae-7412-4b81-a759-77f2355c2055\") "
Sep 10 20:00:58.549776 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.549748    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542b98ae-7412-4b81-a759-77f2355c2055-bundle" (OuterVolumeSpecName: "bundle") pod "542b98ae-7412-4b81-a759-77f2355c2055" (UID: "542b98ae-7412-4b81-a759-77f2355c2055"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:00:58.551479 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.551449    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542b98ae-7412-4b81-a759-77f2355c2055-kube-api-access-rjwvl" (OuterVolumeSpecName: "kube-api-access-rjwvl") pod "542b98ae-7412-4b81-a759-77f2355c2055" (UID: "542b98ae-7412-4b81-a759-77f2355c2055"). InnerVolumeSpecName "kube-api-access-rjwvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:00:58.555176 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.555155    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542b98ae-7412-4b81-a759-77f2355c2055-util" (OuterVolumeSpecName: "util") pod "542b98ae-7412-4b81-a759-77f2355c2055" (UID: "542b98ae-7412-4b81-a759-77f2355c2055"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:00:58.650574 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.650457    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/542b98ae-7412-4b81-a759-77f2355c2055-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:00:58.650574 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.650488    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjwvl\" (UniqueName: \"kubernetes.io/projected/542b98ae-7412-4b81-a759-77f2355c2055-kube-api-access-rjwvl\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:00:58.650574 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:58.650500    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/542b98ae-7412-4b81-a759-77f2355c2055-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:00:59.329004 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.328974    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx"
Sep 10 20:00:59.329453 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.329007    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lh8nx" event={"ID":"542b98ae-7412-4b81-a759-77f2355c2055","Type":"ContainerDied","Data":"19226855efbdad69721f2112f4531f0e59100637a8c6703516be2d797695abb3"}
Sep 10 20:00:59.329453 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.329085    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19226855efbdad69721f2112f4531f0e59100637a8c6703516be2d797695abb3"
Sep 10 20:00:59.467419 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.467391    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"
Sep 10 20:00:59.515405 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.515379    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"
Sep 10 20:00:59.518265 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.518246    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"
Sep 10 20:00:59.558272 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.558241    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdbf8ab5-e278-412d-be46-d33f4a749b72-bundle\") pod \"cdbf8ab5-e278-412d-be46-d33f4a749b72\" (UID: \"cdbf8ab5-e278-412d-be46-d33f4a749b72\") "
Sep 10 20:00:59.558801 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.558349    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdbf8ab5-e278-412d-be46-d33f4a749b72-util\") pod \"cdbf8ab5-e278-412d-be46-d33f4a749b72\" (UID: \"cdbf8ab5-e278-412d-be46-d33f4a749b72\") "
Sep 10 20:00:59.558801 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.558403    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhngq\" (UniqueName: \"kubernetes.io/projected/cdbf8ab5-e278-412d-be46-d33f4a749b72-kube-api-access-lhngq\") pod \"cdbf8ab5-e278-412d-be46-d33f4a749b72\" (UID: \"cdbf8ab5-e278-412d-be46-d33f4a749b72\") "
Sep 10 20:00:59.558801 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.558780    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbf8ab5-e278-412d-be46-d33f4a749b72-bundle" (OuterVolumeSpecName: "bundle") pod "cdbf8ab5-e278-412d-be46-d33f4a749b72" (UID: "cdbf8ab5-e278-412d-be46-d33f4a749b72"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:00:59.559135 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.559113    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdbf8ab5-e278-412d-be46-d33f4a749b72-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:00:59.561837 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.561809    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbf8ab5-e278-412d-be46-d33f4a749b72-kube-api-access-lhngq" (OuterVolumeSpecName: "kube-api-access-lhngq") pod "cdbf8ab5-e278-412d-be46-d33f4a749b72" (UID: "cdbf8ab5-e278-412d-be46-d33f4a749b72"). InnerVolumeSpecName "kube-api-access-lhngq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:00:59.565491 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.565463    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbf8ab5-e278-412d-be46-d33f4a749b72-util" (OuterVolumeSpecName: "util") pod "cdbf8ab5-e278-412d-be46-d33f4a749b72" (UID: "cdbf8ab5-e278-412d-be46-d33f4a749b72"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:00:59.660081 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.659989    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-bundle\") pod \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\" (UID: \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\") "
Sep 10 20:00:59.660081 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.660037    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldvft\" (UniqueName: \"kubernetes.io/projected/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-kube-api-access-ldvft\") pod \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\" (UID: \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\") "
Sep 10 20:00:59.660081 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.660063    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-bundle\") pod \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\" (UID: \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\") "
Sep 10 20:00:59.660375 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.660094    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-util\") pod \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\" (UID: \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\") "
Sep 10 20:00:59.660375 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.660157    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-util\") pod \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\" (UID: \"1557366b-fd3a-4a4c-bceb-cc80f69e7d58\") "
Sep 10 20:00:59.660375 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.660181    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl4mf\" (UniqueName: \"kubernetes.io/projected/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-kube-api-access-jl4mf\") pod \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\" (UID: \"b55d0cd7-7c64-4d67-a7dc-032c1ee06728\") "
Sep 10 20:00:59.660567 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.660394    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdbf8ab5-e278-412d-be46-d33f4a749b72-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:00:59.660567 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.660412    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhngq\" (UniqueName: \"kubernetes.io/projected/cdbf8ab5-e278-412d-be46-d33f4a749b72-kube-api-access-lhngq\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:00:59.660796 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.660741    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-bundle" (OuterVolumeSpecName: "bundle") pod "b55d0cd7-7c64-4d67-a7dc-032c1ee06728" (UID: "b55d0cd7-7c64-4d67-a7dc-032c1ee06728"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:00:59.660909 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.660830    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-bundle" (OuterVolumeSpecName: "bundle") pod "1557366b-fd3a-4a4c-bceb-cc80f69e7d58" (UID: "1557366b-fd3a-4a4c-bceb-cc80f69e7d58"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:00:59.662720 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.662694    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-kube-api-access-ldvft" (OuterVolumeSpecName: "kube-api-access-ldvft") pod "1557366b-fd3a-4a4c-bceb-cc80f69e7d58" (UID: "1557366b-fd3a-4a4c-bceb-cc80f69e7d58"). InnerVolumeSpecName "kube-api-access-ldvft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:00:59.662894 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.662870    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-kube-api-access-jl4mf" (OuterVolumeSpecName: "kube-api-access-jl4mf") pod "b55d0cd7-7c64-4d67-a7dc-032c1ee06728" (UID: "b55d0cd7-7c64-4d67-a7dc-032c1ee06728"). InnerVolumeSpecName "kube-api-access-jl4mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:00:59.666839 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.666817    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-util" (OuterVolumeSpecName: "util") pod "b55d0cd7-7c64-4d67-a7dc-032c1ee06728" (UID: "b55d0cd7-7c64-4d67-a7dc-032c1ee06728"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:00:59.666903 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.666815    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-util" (OuterVolumeSpecName: "util") pod "1557366b-fd3a-4a4c-bceb-cc80f69e7d58" (UID: "1557366b-fd3a-4a4c-bceb-cc80f69e7d58"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:00:59.761651 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.761616    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:00:59.761651 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.761644    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:00:59.761651 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.761653    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:00:59.761894 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.761661    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl4mf\" (UniqueName: \"kubernetes.io/projected/b55d0cd7-7c64-4d67-a7dc-032c1ee06728-kube-api-access-jl4mf\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:00:59.761894 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.761671    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:00:59.761894 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:00:59.761679    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldvft\" (UniqueName: \"kubernetes.io/projected/1557366b-fd3a-4a4c-bceb-cc80f69e7d58-kube-api-access-ldvft\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:01:00.336605 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:00.336578    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd"
Sep 10 20:01:00.337022 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:00.336571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjlrd" event={"ID":"cdbf8ab5-e278-412d-be46-d33f4a749b72","Type":"ContainerDied","Data":"a07c30bb74fe939a0b426b2b10d2c8881247fccfa49b33820c1a5b31ffec112f"}
Sep 10 20:01:00.337022 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:00.336710    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a07c30bb74fe939a0b426b2b10d2c8881247fccfa49b33820c1a5b31ffec112f"
Sep 10 20:01:00.338618 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:00.338599    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx"
Sep 10 20:01:00.338729 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:00.338636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xxkx" event={"ID":"1557366b-fd3a-4a4c-bceb-cc80f69e7d58","Type":"ContainerDied","Data":"7ceaa0cb8eede6a10cab969adef341d61da1ddb543866288145e39306985622a"}
Sep 10 20:01:00.338729 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:00.338676    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ceaa0cb8eede6a10cab969adef341d61da1ddb543866288145e39306985622a"
Sep 10 20:01:00.340798 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:00.340775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f" event={"ID":"b55d0cd7-7c64-4d67-a7dc-032c1ee06728","Type":"ContainerDied","Data":"e2a8ac6fb16765d79fa6a4f837a6cd1ad34a66bf880b0095d73632756cfa2ffc"}
Sep 10 20:01:00.340888 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:00.340817    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2a8ac6fb16765d79fa6a4f837a6cd1ad34a66bf880b0095d73632756cfa2ffc"
Sep 10 20:01:00.340888 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:00.340820    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6cd6f"
Sep 10 20:01:05.019944 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:05.019911    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:06.396309 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:06.396273    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:06.550336 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:06.550296    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:06.654396 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:06.654317    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:07.143923 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:07.143891    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:07.235208 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:07.235158    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:07.879657 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:07.879624    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:08.235699 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:08.235621    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:08.645278 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:08.645244    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:09.656580 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:09.656521    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:11.272445 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:11.272414    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:11.409130 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:11.409093    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:12.020755 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:12.020726    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:12.437210 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:12.437129    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:12.693124 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:12.693026    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:15.009974 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:15.009941    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:15.443513 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:15.443480    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:15.542026 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:15.541993    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:16.871363 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:16.871312    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:17.817426 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:17.817383    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:17.920119 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:17.920086    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:18.649184 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:18.649148    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:18.749220 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:18.749188    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:22.692036 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:22.691999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:23.046762 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:23.046734    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:23.611709 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:23.611673    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:23.719496 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:23.719461    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:24.225403 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:24.225360    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:24.339463 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:24.339432    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:24.806922 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:24.806888    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:24.927188 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:24.927159    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:32.345031 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:32.344997    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:32.524085 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:32.524052    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:32.708767 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:32.708675    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:35.249004 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:35.248961    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:35.361586 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:35.361559    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:38.088953 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:38.088916    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:38.367947 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:38.367860    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:42.696548 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:42.696458    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:42.802380 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:42.802347    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:42.907184 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:42.907144    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:51.513553 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:51.513508    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:51.622352 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:51.622316    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:55.729327 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:55.729282    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:55.978270 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:55.978234    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:57.139033 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:57.138999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:01:57.243441 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:01:57.243409    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:02.552216 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:02.552180    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:02.669336 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:02.669305    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:02.946687 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:02.946605    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:03.018039 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:03.018009    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:03.328278 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:03.328228    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:03.400004 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:03.399969    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:06.509258 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:06.509227    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:06.752287 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:06.752259    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:10.988070 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:10.988041    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:11.092279 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.092249    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:11.163370 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.163334    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:11.379806 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.379773    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:11.460877 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.460845    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:11.833347 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833308    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hbqwr"]
Sep 10 20:02:11.833660 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833647    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbf8ab5-e278-412d-be46-d33f4a749b72" containerName="pull"
Sep 10 20:02:11.833660 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833662    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbf8ab5-e278-412d-be46-d33f4a749b72" containerName="pull"
Sep 10 20:02:11.833743 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833671    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b55d0cd7-7c64-4d67-a7dc-032c1ee06728" containerName="extract"
Sep 10 20:02:11.833743 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833677    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b55d0cd7-7c64-4d67-a7dc-032c1ee06728" containerName="extract"
Sep 10 20:02:11.833743 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833684    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbf8ab5-e278-412d-be46-d33f4a749b72" containerName="extract"
Sep 10 20:02:11.833743 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833690    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbf8ab5-e278-412d-be46-d33f4a749b72" containerName="extract"
Sep 10 20:02:11.833743 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833698    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="542b98ae-7412-4b81-a759-77f2355c2055" containerName="util"
Sep 10 20:02:11.833743 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833703    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="542b98ae-7412-4b81-a759-77f2355c2055" containerName="util"
Sep 10 20:02:11.833743 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833709    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="542b98ae-7412-4b81-a759-77f2355c2055" containerName="pull"
Sep 10 20:02:11.833743 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833713    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="542b98ae-7412-4b81-a759-77f2355c2055" containerName="pull"
Sep 10 20:02:11.833743 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833723    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1557366b-fd3a-4a4c-bceb-cc80f69e7d58" containerName="util"
Sep 10 20:02:11.833743 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833728    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1557366b-fd3a-4a4c-bceb-cc80f69e7d58" containerName="util"
Sep 10 20:02:11.833743 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833733    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1557366b-fd3a-4a4c-bceb-cc80f69e7d58" containerName="extract"
Sep 10 20:02:11.833743 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833738    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1557366b-fd3a-4a4c-bceb-cc80f69e7d58" containerName="extract"
Sep 10 20:02:11.834080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833749    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1557366b-fd3a-4a4c-bceb-cc80f69e7d58" containerName="pull"
Sep 10 20:02:11.834080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833754    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1557366b-fd3a-4a4c-bceb-cc80f69e7d58" containerName="pull"
Sep 10 20:02:11.834080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833761    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbf8ab5-e278-412d-be46-d33f4a749b72" containerName="util"
Sep 10 20:02:11.834080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833766    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbf8ab5-e278-412d-be46-d33f4a749b72" containerName="util"
Sep 10 20:02:11.834080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833773    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b55d0cd7-7c64-4d67-a7dc-032c1ee06728" containerName="pull"
Sep 10 20:02:11.834080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833778    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b55d0cd7-7c64-4d67-a7dc-032c1ee06728" containerName="pull"
Sep 10 20:02:11.834080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833784    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b55d0cd7-7c64-4d67-a7dc-032c1ee06728" containerName="util"
Sep 10 20:02:11.834080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833789    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b55d0cd7-7c64-4d67-a7dc-032c1ee06728" containerName="util"
Sep 10 20:02:11.834080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833795    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="542b98ae-7412-4b81-a759-77f2355c2055" containerName="extract"
Sep 10 20:02:11.834080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833799    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="542b98ae-7412-4b81-a759-77f2355c2055" containerName="extract"
Sep 10 20:02:11.834080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833845    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1557366b-fd3a-4a4c-bceb-cc80f69e7d58" containerName="extract"
Sep 10 20:02:11.834080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833855    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="542b98ae-7412-4b81-a759-77f2355c2055" containerName="extract"
Sep 10 20:02:11.834080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833862    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b55d0cd7-7c64-4d67-a7dc-032c1ee06728" containerName="extract"
Sep 10 20:02:11.834080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.833868    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdbf8ab5-e278-412d-be46-d33f4a749b72" containerName="extract"
Sep 10 20:02:11.836701 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.836682    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hbqwr"
Sep 10 20:02:11.839072 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.839045    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6n2dz\""
Sep 10 20:02:11.839180 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.839049    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 20:02:11.840000 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.839983    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 20:02:11.854885 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.854861    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hbqwr"]
Sep 10 20:02:11.875892 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.875861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfwzd\" (UniqueName: \"kubernetes.io/projected/7b0141a7-590a-4005-b744-1b3a17984472-kube-api-access-rfwzd\") pod \"authorino-7bcfd8f9cd-hbqwr\" (UID: \"7b0141a7-590a-4005-b744-1b3a17984472\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hbqwr"
Sep 10 20:02:11.949166 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.949131    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:11.976285 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.976254    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfwzd\" (UniqueName: \"kubernetes.io/projected/7b0141a7-590a-4005-b744-1b3a17984472-kube-api-access-rfwzd\") pod \"authorino-7bcfd8f9cd-hbqwr\" (UID: \"7b0141a7-590a-4005-b744-1b3a17984472\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hbqwr"
Sep 10 20:02:11.998438 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:11.998398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfwzd\" (UniqueName: \"kubernetes.io/projected/7b0141a7-590a-4005-b744-1b3a17984472-kube-api-access-rfwzd\") pod \"authorino-7bcfd8f9cd-hbqwr\" (UID: \"7b0141a7-590a-4005-b744-1b3a17984472\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hbqwr"
Sep 10 20:02:12.120092 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:12.120001    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:12.146012 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:12.145975    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hbqwr"
Sep 10 20:02:12.438459 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:12.438435    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hbqwr"]
Sep 10 20:02:12.440873 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:02:12.440841    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b0141a7_590a_4005_b744_1b3a17984472.slice/crio-765c938c243f69a9e3358c08cd0229ee00f34347e5173a1d8ef00bd208afe5d1 WatchSource:0}: Error finding container 765c938c243f69a9e3358c08cd0229ee00f34347e5173a1d8ef00bd208afe5d1: Status 404 returned error can't find the container with id 765c938c243f69a9e3358c08cd0229ee00f34347e5173a1d8ef00bd208afe5d1
Sep 10 20:02:12.698057 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:12.697969    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hbqwr" event={"ID":"7b0141a7-590a-4005-b744-1b3a17984472","Type":"ContainerStarted","Data":"765c938c243f69a9e3358c08cd0229ee00f34347e5173a1d8ef00bd208afe5d1"}
Sep 10 20:02:14.395709 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:14.395661    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:14.496469 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:14.496427    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:15.941324 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:15.941302    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hbqwr"]
Sep 10 20:02:16.722028 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:16.721995    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hbqwr" event={"ID":"7b0141a7-590a-4005-b744-1b3a17984472","Type":"ContainerStarted","Data":"cf051cb9bcd3145f6a3a779834c7bb8b3310a6745feeec6323183619ce847e9a"}
Sep 10 20:02:16.722338 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:16.722061    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-hbqwr" podUID="7b0141a7-590a-4005-b744-1b3a17984472" containerName="authorino" containerID="cri-o://cf051cb9bcd3145f6a3a779834c7bb8b3310a6745feeec6323183619ce847e9a" gracePeriod=30
Sep 10 20:02:16.739421 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:16.739377    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-hbqwr" podStartSLOduration=2.258443614 podStartE2EDuration="5.739362685s" totalImagesPullingTime="3.480919071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:02:16.737285262 +0000 UTC m=+477.787813251" watchObservedRunningTime="2026-09-10 20:02:16.739362685 +0000 UTC m=+477.789890674"
Sep 10 20:02:16.967317 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:16.967293    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hbqwr"
Sep 10 20:02:17.021759 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:17.021725    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfwzd\" (UniqueName: \"kubernetes.io/projected/7b0141a7-590a-4005-b744-1b3a17984472-kube-api-access-rfwzd\") pod \"7b0141a7-590a-4005-b744-1b3a17984472\" (UID: \"7b0141a7-590a-4005-b744-1b3a17984472\") "
Sep 10 20:02:17.024073 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:17.024043    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0141a7-590a-4005-b744-1b3a17984472-kube-api-access-rfwzd" (OuterVolumeSpecName: "kube-api-access-rfwzd") pod "7b0141a7-590a-4005-b744-1b3a17984472" (UID: "7b0141a7-590a-4005-b744-1b3a17984472"). InnerVolumeSpecName "kube-api-access-rfwzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:02:17.122757 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:17.122716    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfwzd\" (UniqueName: \"kubernetes.io/projected/7b0141a7-590a-4005-b744-1b3a17984472-kube-api-access-rfwzd\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:02:17.728754 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:17.728657    2561 generic.go:356] "Generic (PLEG): container finished" podID="7b0141a7-590a-4005-b744-1b3a17984472" containerID="cf051cb9bcd3145f6a3a779834c7bb8b3310a6745feeec6323183619ce847e9a" exitCode=0
Sep 10 20:02:17.728754 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:17.728717    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hbqwr"
Sep 10 20:02:17.728754 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:17.728740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hbqwr" event={"ID":"7b0141a7-590a-4005-b744-1b3a17984472","Type":"ContainerDied","Data":"cf051cb9bcd3145f6a3a779834c7bb8b3310a6745feeec6323183619ce847e9a"}
Sep 10 20:02:17.729037 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:17.728785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hbqwr" event={"ID":"7b0141a7-590a-4005-b744-1b3a17984472","Type":"ContainerDied","Data":"765c938c243f69a9e3358c08cd0229ee00f34347e5173a1d8ef00bd208afe5d1"}
Sep 10 20:02:17.729037 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:17.728803    2561 scope.go:117] "RemoveContainer" containerID="cf051cb9bcd3145f6a3a779834c7bb8b3310a6745feeec6323183619ce847e9a"
Sep 10 20:02:17.737669 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:17.737647    2561 scope.go:117] "RemoveContainer" containerID="cf051cb9bcd3145f6a3a779834c7bb8b3310a6745feeec6323183619ce847e9a"
Sep 10 20:02:17.737977 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:02:17.737957    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf051cb9bcd3145f6a3a779834c7bb8b3310a6745feeec6323183619ce847e9a\": container with ID starting with cf051cb9bcd3145f6a3a779834c7bb8b3310a6745feeec6323183619ce847e9a not found: ID does not exist" containerID="cf051cb9bcd3145f6a3a779834c7bb8b3310a6745feeec6323183619ce847e9a"
Sep 10 20:02:17.738061 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:17.737984    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf051cb9bcd3145f6a3a779834c7bb8b3310a6745feeec6323183619ce847e9a"} err="failed to get container status \"cf051cb9bcd3145f6a3a779834c7bb8b3310a6745feeec6323183619ce847e9a\": rpc error: code = NotFound desc = could not find container \"cf051cb9bcd3145f6a3a779834c7bb8b3310a6745feeec6323183619ce847e9a\": container with ID starting with cf051cb9bcd3145f6a3a779834c7bb8b3310a6745feeec6323183619ce847e9a not found: ID does not exist"
Sep 10 20:02:17.746522 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:17.746495    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hbqwr"]
Sep 10 20:02:17.749330 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:17.749311    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hbqwr"]
Sep 10 20:02:19.403721 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:19.403686    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b0141a7-590a-4005-b744-1b3a17984472" path="/var/lib/kubelet/pods/7b0141a7-590a-4005-b744-1b3a17984472/volumes"
Sep 10 20:02:27.133797 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:27.133753    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:27.220850 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:27.220818    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:27.641860 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:27.641827    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:37.405131 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.405098    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:37.632759 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.629781    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-69km9"]
Sep 10 20:02:37.632759 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.630458    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0141a7-590a-4005-b744-1b3a17984472" containerName="authorino"
Sep 10 20:02:37.632759 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.630477    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0141a7-590a-4005-b744-1b3a17984472" containerName="authorino"
Sep 10 20:02:37.632759 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.630680    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b0141a7-590a-4005-b744-1b3a17984472" containerName="authorino"
Sep 10 20:02:37.635918 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.635893    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-69km9"
Sep 10 20:02:37.639067 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.638870    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 20:02:37.639891 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.639870    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6n2dz\""
Sep 10 20:02:37.641807 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.641782    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 20:02:37.641929 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.641850    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-69km9"]
Sep 10 20:02:37.684352 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.684275    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:37.699496 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.699460    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l5gv\" (UniqueName: \"kubernetes.io/projected/f5a604a3-f122-416c-adcd-1e99d4144375-kube-api-access-9l5gv\") pod \"authorino-557f876f69-69km9\" (UID: \"f5a604a3-f122-416c-adcd-1e99d4144375\") " pod="kuadrant-system/authorino-557f876f69-69km9"
Sep 10 20:02:37.791428 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.791391    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-69km9"]
Sep 10 20:02:37.791682 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:02:37.791663    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9l5gv], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-69km9" podUID="f5a604a3-f122-416c-adcd-1e99d4144375"
Sep 10 20:02:37.800501 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.800474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5gv\" (UniqueName: \"kubernetes.io/projected/f5a604a3-f122-416c-adcd-1e99d4144375-kube-api-access-9l5gv\") pod \"authorino-557f876f69-69km9\" (UID: \"f5a604a3-f122-416c-adcd-1e99d4144375\") " pod="kuadrant-system/authorino-557f876f69-69km9"
Sep 10 20:02:37.813147 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.813110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l5gv\" (UniqueName: \"kubernetes.io/projected/f5a604a3-f122-416c-adcd-1e99d4144375-kube-api-access-9l5gv\") pod \"authorino-557f876f69-69km9\" (UID: \"f5a604a3-f122-416c-adcd-1e99d4144375\") " pod="kuadrant-system/authorino-557f876f69-69km9"
Sep 10 20:02:37.832087 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.832055    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-69km9"
Sep 10 20:02:37.839411 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.839387    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-69km9"
Sep 10 20:02:37.901253 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.901219    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l5gv\" (UniqueName: \"kubernetes.io/projected/f5a604a3-f122-416c-adcd-1e99d4144375-kube-api-access-9l5gv\") pod \"f5a604a3-f122-416c-adcd-1e99d4144375\" (UID: \"f5a604a3-f122-416c-adcd-1e99d4144375\") "
Sep 10 20:02:37.903641 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.903612    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a604a3-f122-416c-adcd-1e99d4144375-kube-api-access-9l5gv" (OuterVolumeSpecName: "kube-api-access-9l5gv") pod "f5a604a3-f122-416c-adcd-1e99d4144375" (UID: "f5a604a3-f122-416c-adcd-1e99d4144375"). InnerVolumeSpecName "kube-api-access-9l5gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:02:37.962654 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.962575    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c4c6c779b-x5sz6"]
Sep 10 20:02:37.965839 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.965820    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4c6c779b-x5sz6"
Sep 10 20:02:37.968639 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.968617    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 20:02:37.975515 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:37.975488    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c4c6c779b-x5sz6"]
Sep 10 20:02:38.002223 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:38.002181    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l5gv\" (UniqueName: \"kubernetes.io/projected/f5a604a3-f122-416c-adcd-1e99d4144375-kube-api-access-9l5gv\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:02:38.103520 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:38.103475    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trlws\" (UniqueName: \"kubernetes.io/projected/ade89cc5-285b-4b69-aa58-bd312c3a1453-kube-api-access-trlws\") pod \"authorino-6c4c6c779b-x5sz6\" (UID: \"ade89cc5-285b-4b69-aa58-bd312c3a1453\") " pod="kuadrant-system/authorino-6c4c6c779b-x5sz6"
Sep 10 20:02:38.103520 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:38.103519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ade89cc5-285b-4b69-aa58-bd312c3a1453-tls-cert\") pod \"authorino-6c4c6c779b-x5sz6\" (UID: \"ade89cc5-285b-4b69-aa58-bd312c3a1453\") " pod="kuadrant-system/authorino-6c4c6c779b-x5sz6"
Sep 10 20:02:38.204882 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:38.204848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trlws\" (UniqueName: \"kubernetes.io/projected/ade89cc5-285b-4b69-aa58-bd312c3a1453-kube-api-access-trlws\") pod \"authorino-6c4c6c779b-x5sz6\" (UID: \"ade89cc5-285b-4b69-aa58-bd312c3a1453\") " pod="kuadrant-system/authorino-6c4c6c779b-x5sz6"
Sep 10 20:02:38.204882 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:38.204888    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ade89cc5-285b-4b69-aa58-bd312c3a1453-tls-cert\") pod \"authorino-6c4c6c779b-x5sz6\" (UID: \"ade89cc5-285b-4b69-aa58-bd312c3a1453\") " pod="kuadrant-system/authorino-6c4c6c779b-x5sz6"
Sep 10 20:02:38.207510 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:38.207491    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ade89cc5-285b-4b69-aa58-bd312c3a1453-tls-cert\") pod \"authorino-6c4c6c779b-x5sz6\" (UID: \"ade89cc5-285b-4b69-aa58-bd312c3a1453\") " pod="kuadrant-system/authorino-6c4c6c779b-x5sz6"
Sep 10 20:02:38.215951 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:38.215896    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trlws\" (UniqueName: \"kubernetes.io/projected/ade89cc5-285b-4b69-aa58-bd312c3a1453-kube-api-access-trlws\") pod \"authorino-6c4c6c779b-x5sz6\" (UID: \"ade89cc5-285b-4b69-aa58-bd312c3a1453\") " pod="kuadrant-system/authorino-6c4c6c779b-x5sz6"
Sep 10 20:02:38.275727 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:38.275687    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4c6c779b-x5sz6"
Sep 10 20:02:38.553419 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:38.553395    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c4c6c779b-x5sz6"]
Sep 10 20:02:38.554790 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:02:38.554770    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podade89cc5_285b_4b69_aa58_bd312c3a1453.slice/crio-006455e041ce1ac2c53451184113536142c4f3ea210b8b22fcf4490541cbafc9 WatchSource:0}: Error finding container 006455e041ce1ac2c53451184113536142c4f3ea210b8b22fcf4490541cbafc9: Status 404 returned error can't find the container with id 006455e041ce1ac2c53451184113536142c4f3ea210b8b22fcf4490541cbafc9
Sep 10 20:02:38.838291 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:38.838205    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-69km9"
Sep 10 20:02:38.838291 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:38.838228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c4c6c779b-x5sz6" event={"ID":"ade89cc5-285b-4b69-aa58-bd312c3a1453","Type":"ContainerStarted","Data":"006455e041ce1ac2c53451184113536142c4f3ea210b8b22fcf4490541cbafc9"}
Sep 10 20:02:38.872615 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:38.872580    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-69km9"]
Sep 10 20:02:38.874545 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:38.874499    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-69km9"]
Sep 10 20:02:39.404079 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:39.403999    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5a604a3-f122-416c-adcd-1e99d4144375" path="/var/lib/kubelet/pods/f5a604a3-f122-416c-adcd-1e99d4144375/volumes"
Sep 10 20:02:39.848370 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:39.848334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c4c6c779b-x5sz6" event={"ID":"ade89cc5-285b-4b69-aa58-bd312c3a1453","Type":"ContainerStarted","Data":"a6792ece76efbacf6b338d70492bcf0af709bf8c6d99b1340f8942c982701ba8"}
Sep 10 20:02:39.873860 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:39.873809    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c4c6c779b-x5sz6" podStartSLOduration=2.295565608 podStartE2EDuration="2.873793642s" totalImagesPullingTime="578.228034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:02:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:02:39.873217773 +0000 UTC m=+500.923745763" watchObservedRunningTime="2026-09-10 20:02:39.873793642 +0000 UTC m=+500.924321631"
Sep 10 20:02:45.346834 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:45.346798    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:45.446451 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:45.446419    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:45.548096 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:45.548062    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:45.626241 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:45.626149    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:46.263859 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:46.263825    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:49.647565 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:49.647513    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:49.771018 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:49.770985    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:02:52.351738 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:02:52.351704    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:03:05.570266 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:05.570224    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:03:22.430952 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:22.430859    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:03:22.529623 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:22.529594    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:03:23.905720 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:23.905684    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:03:23.979442 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:23.979407    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"]
Sep 10 20:03:23.981598 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:23.981569    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:03:23.982423 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:23.982407    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:23.986334 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:23.986308    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 20:03:23.986493 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:23.986336    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 20:03:23.986493 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:23.986314    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 20:03:23.986493 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:23.986346    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 20:03:23.987334 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:23.987320    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q7b27\""
Sep 10 20:03:23.995407 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:23.995369    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"]
Sep 10 20:03:24.072088 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.072058    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-maas-api-tls\") pod \"maas-api-7f7f8945f9-qt2rc\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:24.072248 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.072104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tnfl\" (UniqueName: \"kubernetes.io/projected/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-kube-api-access-4tnfl\") pod \"maas-api-7f7f8945f9-qt2rc\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:24.072248 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.072158    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-metrics-certs\") pod \"maas-api-7f7f8945f9-qt2rc\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:24.072248 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.072225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-openshift-service-ca\") pod \"maas-api-7f7f8945f9-qt2rc\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:24.173632 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.173506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-openshift-service-ca\") pod \"maas-api-7f7f8945f9-qt2rc\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:24.173784 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.173634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-maas-api-tls\") pod \"maas-api-7f7f8945f9-qt2rc\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:24.173784 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.173686    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tnfl\" (UniqueName: \"kubernetes.io/projected/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-kube-api-access-4tnfl\") pod \"maas-api-7f7f8945f9-qt2rc\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:24.173854 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:03:24.173776    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 10 20:03:24.173888 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:03:24.173864    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-maas-api-tls podName:f0cbd8c5-b68d-47dc-8339-d012dc5bfb65 nodeName:}" failed. No retries permitted until 2026-09-10 20:03:24.673843216 +0000 UTC m=+545.724371190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-maas-api-tls") pod "maas-api-7f7f8945f9-qt2rc" (UID: "f0cbd8c5-b68d-47dc-8339-d012dc5bfb65") : secret "maas-api-serving-cert" not found
Sep 10 20:03:24.173933 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.173885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-metrics-certs\") pod \"maas-api-7f7f8945f9-qt2rc\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:24.173978 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:03:24.173966    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 10 20:03:24.174013 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:03:24.174009    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-metrics-certs podName:f0cbd8c5-b68d-47dc-8339-d012dc5bfb65 nodeName:}" failed. No retries permitted until 2026-09-10 20:03:24.673997125 +0000 UTC m=+545.724525092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-metrics-certs") pod "maas-api-7f7f8945f9-qt2rc" (UID: "f0cbd8c5-b68d-47dc-8339-d012dc5bfb65") : secret "maas-api-metrics-tls" not found
Sep 10 20:03:24.174255 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.174236    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-openshift-service-ca\") pod \"maas-api-7f7f8945f9-qt2rc\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:24.184969 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.184934    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tnfl\" (UniqueName: \"kubernetes.io/projected/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-kube-api-access-4tnfl\") pod \"maas-api-7f7f8945f9-qt2rc\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:24.270282 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.270254    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:03:24.396577 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.396486    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:03:24.503211 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.503133    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:03:24.679583 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.679546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-maas-api-tls\") pod \"maas-api-7f7f8945f9-qt2rc\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:24.679782 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.679624    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-metrics-certs\") pod \"maas-api-7f7f8945f9-qt2rc\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:24.682195 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.682161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-metrics-certs\") pod \"maas-api-7f7f8945f9-qt2rc\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:24.682195 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.682174    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-maas-api-tls\") pod \"maas-api-7f7f8945f9-qt2rc\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:24.893592 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:24.893505    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:25.190670 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:25.190641    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"]
Sep 10 20:03:25.194988 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:03:25.194912    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0cbd8c5_b68d_47dc_8339_d012dc5bfb65.slice/crio-424e9bb9aac05e2b92ca7ebdb8cf10af537bf7c690d5b111587352b2b22f402c WatchSource:0}: Error finding container 424e9bb9aac05e2b92ca7ebdb8cf10af537bf7c690d5b111587352b2b22f402c: Status 404 returned error can't find the container with id 424e9bb9aac05e2b92ca7ebdb8cf10af537bf7c690d5b111587352b2b22f402c
Sep 10 20:03:26.083710 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:26.083673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc" event={"ID":"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65","Type":"ContainerStarted","Data":"424e9bb9aac05e2b92ca7ebdb8cf10af537bf7c690d5b111587352b2b22f402c"}
Sep 10 20:03:28.097999 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:28.097961    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc" event={"ID":"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65","Type":"ContainerStarted","Data":"0b8364da3843607b8c1408c1f544300d9908a9b8887edd18d8aeae7c83906710"}
Sep 10 20:03:28.098393 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:28.098071    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:28.120003 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:28.119954    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc" podStartSLOduration=2.384702952 podStartE2EDuration="5.119936345s" totalImagesPullingTime="2.735233393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:03:28.118064968 +0000 UTC m=+549.168592968" watchObservedRunningTime="2026-09-10 20:03:28.119936345 +0000 UTC m=+549.170464333"
Sep 10 20:03:34.109239 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:34.109204    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:03:35.107332 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:35.107290    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:03:35.208520 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:35.208478    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:03:42.885838 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:42.885797    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:03:43.093545 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:43.093498    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:03:43.197073 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:43.196972    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:03:56.482593 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:03:56.482556    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:04:05.569292 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:05.569260    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:04:15.378901 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:15.378866    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:04:20.770494 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:20.770467    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 20:04:20.772169 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:20.771978    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 20:04:45.055410 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.055333    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"]
Sep 10 20:04:45.057564 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.057548    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.060974 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.060946    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 20:04:45.061110 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.060950    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j5zgl\""
Sep 10 20:04:45.061110 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.060985    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 20:04:45.061110 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.061035    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 20:04:45.069293 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.069269    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"]
Sep 10 20:04:45.097960 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.097929    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:04:45.184823 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.184780    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6f9114-2eff-4720-83a6-f98263aaabd4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.184995 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.184832    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6f9114-2eff-4720-83a6-f98263aaabd4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.184995 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.184946    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6f9114-2eff-4720-83a6-f98263aaabd4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.184995 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.184985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6f9114-2eff-4720-83a6-f98263aaabd4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.185149 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.185015    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52gmx\" (UniqueName: \"kubernetes.io/projected/3a6f9114-2eff-4720-83a6-f98263aaabd4-kube-api-access-52gmx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.185149 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.185093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6f9114-2eff-4720-83a6-f98263aaabd4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.286278 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.286241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6f9114-2eff-4720-83a6-f98263aaabd4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.286418 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.286309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6f9114-2eff-4720-83a6-f98263aaabd4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.286418 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.286329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6f9114-2eff-4720-83a6-f98263aaabd4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.286418 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.286349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52gmx\" (UniqueName: \"kubernetes.io/projected/3a6f9114-2eff-4720-83a6-f98263aaabd4-kube-api-access-52gmx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.286418 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.286388    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6f9114-2eff-4720-83a6-f98263aaabd4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.286665 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.286521    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6f9114-2eff-4720-83a6-f98263aaabd4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.286804 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.286783    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6f9114-2eff-4720-83a6-f98263aaabd4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.286860 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.286799    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6f9114-2eff-4720-83a6-f98263aaabd4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.286898 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.286870    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6f9114-2eff-4720-83a6-f98263aaabd4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.288849 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.288827    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6f9114-2eff-4720-83a6-f98263aaabd4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.289069 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.289054    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6f9114-2eff-4720-83a6-f98263aaabd4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.294043 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.294022    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52gmx\" (UniqueName: \"kubernetes.io/projected/3a6f9114-2eff-4720-83a6-f98263aaabd4-kube-api-access-52gmx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6\" (UID: \"3a6f9114-2eff-4720-83a6-f98263aaabd4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.369274 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.369180    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:45.642903 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.642878    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"]
Sep 10 20:04:45.644563 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:04:45.644519    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6f9114_2eff_4720_83a6_f98263aaabd4.slice/crio-978436095e010b7dff16329149e8390dd34f9cd746ba67564afa2f3c86eb535b WatchSource:0}: Error finding container 978436095e010b7dff16329149e8390dd34f9cd746ba67564afa2f3c86eb535b: Status 404 returned error can't find the container with id 978436095e010b7dff16329149e8390dd34f9cd746ba67564afa2f3c86eb535b
Sep 10 20:04:45.645966 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:45.645946    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:04:46.501596 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:46.501562    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6" event={"ID":"3a6f9114-2eff-4720-83a6-f98263aaabd4","Type":"ContainerStarted","Data":"978436095e010b7dff16329149e8390dd34f9cd746ba67564afa2f3c86eb535b"}
Sep 10 20:04:47.508922 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:47.508890    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6" event={"ID":"3a6f9114-2eff-4720-83a6-f98263aaabd4","Type":"ContainerStarted","Data":"16121b2ff9d19f41fe98a964ca723487181d81f4d64414e490028b871eae8a78"}
Sep 10 20:04:47.527403 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:47.527350    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6" podStartSLOduration=1.526343377 podStartE2EDuration="2.527330232s" totalImagesPullingTime="1.000986855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:04:47.525334699 +0000 UTC m=+628.575862694" watchObservedRunningTime="2026-09-10 20:04:47.527330232 +0000 UTC m=+628.577858221"
Sep 10 20:04:52.957196 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:52.957160    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:04:55.372241 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:55.372196    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:55.374815 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:55.374776    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:55.387097 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:55.387068    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:55.562492 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:55.562464    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6"
Sep 10 20:04:56.441075 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.441039    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"]
Sep 10 20:04:56.443601 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.443572    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.445929 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.445909    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 20:04:56.456153 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.456132    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"]
Sep 10 20:04:56.484502 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.484459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08530dcd-3b4c-4b11-b6af-32898f41fc84-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.484721 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.484652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08530dcd-3b4c-4b11-b6af-32898f41fc84-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.484721 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.484693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb5vr\" (UniqueName: \"kubernetes.io/projected/08530dcd-3b4c-4b11-b6af-32898f41fc84-kube-api-access-rb5vr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.484851 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.484728    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08530dcd-3b4c-4b11-b6af-32898f41fc84-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.484922 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.484899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08530dcd-3b4c-4b11-b6af-32898f41fc84-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.484976 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.484936    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08530dcd-3b4c-4b11-b6af-32898f41fc84-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.561851 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.561819    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:04:56.585349 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.585310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08530dcd-3b4c-4b11-b6af-32898f41fc84-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.585349 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.585349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08530dcd-3b4c-4b11-b6af-32898f41fc84-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.585625 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.585404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08530dcd-3b4c-4b11-b6af-32898f41fc84-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.585625 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.585434    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08530dcd-3b4c-4b11-b6af-32898f41fc84-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.585625 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.585461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb5vr\" (UniqueName: \"kubernetes.io/projected/08530dcd-3b4c-4b11-b6af-32898f41fc84-kube-api-access-rb5vr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.585625 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.585504    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08530dcd-3b4c-4b11-b6af-32898f41fc84-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.586073 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.585776    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08530dcd-3b4c-4b11-b6af-32898f41fc84-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.586073 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.585920    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08530dcd-3b4c-4b11-b6af-32898f41fc84-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.586073 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.585971    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08530dcd-3b4c-4b11-b6af-32898f41fc84-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.587800 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.587779    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08530dcd-3b4c-4b11-b6af-32898f41fc84-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.588102 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.588085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08530dcd-3b4c-4b11-b6af-32898f41fc84-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.593650 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.593631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb5vr\" (UniqueName: \"kubernetes.io/projected/08530dcd-3b4c-4b11-b6af-32898f41fc84-kube-api-access-rb5vr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g58br\" (UID: \"08530dcd-3b4c-4b11-b6af-32898f41fc84\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:56.751832 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:56.751746    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:04:57.025655 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:57.025631    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"]
Sep 10 20:04:57.027675 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:04:57.027652    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08530dcd_3b4c_4b11_b6af_32898f41fc84.slice/crio-8f421e365cffeec7d4b3ffbb71a9e2b4da3bb9db4d8ac7d1ae79644d394b9296 WatchSource:0}: Error finding container 8f421e365cffeec7d4b3ffbb71a9e2b4da3bb9db4d8ac7d1ae79644d394b9296: Status 404 returned error can't find the container with id 8f421e365cffeec7d4b3ffbb71a9e2b4da3bb9db4d8ac7d1ae79644d394b9296
Sep 10 20:04:57.562357 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:57.562255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br" event={"ID":"08530dcd-3b4c-4b11-b6af-32898f41fc84","Type":"ContainerStarted","Data":"cf507dd5d8ef855bb8b9f2697c0846da280682d4c900e33a558f5da46f2fecd5"}
Sep 10 20:04:57.562357 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:57.562305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br" event={"ID":"08530dcd-3b4c-4b11-b6af-32898f41fc84","Type":"ContainerStarted","Data":"8f421e365cffeec7d4b3ffbb71a9e2b4da3bb9db4d8ac7d1ae79644d394b9296"}
Sep 10 20:04:57.579184 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:57.579135    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br" podStartSLOduration=1.349749104 podStartE2EDuration="1.579120481s" totalImagesPullingTime="229.371377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:04:57.578111839 +0000 UTC m=+638.628639828" watchObservedRunningTime="2026-09-10 20:04:57.579120481 +0000 UTC m=+638.629648469"
Sep 10 20:04:58.643478 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:58.643451    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:04:58.744917 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:04:58.744877    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:05:06.752736 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:05:06.752690    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:05:06.753259 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:05:06.752771    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:05:06.765749 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:05:06.765715    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:05:07.633423 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:05:07.633393    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g58br"
Sep 10 20:05:44.141893 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:05:44.141861    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:05:48.062000 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:05:48.061965    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:05:48.113888 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:05:48.113794    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:28.141200 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:28.141119    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c4c6c779b-x5sz6"]
Sep 10 20:06:28.141758 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:28.141307    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6c4c6c779b-x5sz6" podUID="ade89cc5-285b-4b69-aa58-bd312c3a1453" containerName="authorino" containerID="cri-o://a6792ece76efbacf6b338d70492bcf0af709bf8c6d99b1340f8942c982701ba8" gracePeriod=30
Sep 10 20:06:28.385963 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:28.385940    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4c6c779b-x5sz6"
Sep 10 20:06:28.494209 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:28.494109    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ade89cc5-285b-4b69-aa58-bd312c3a1453-tls-cert\") pod \"ade89cc5-285b-4b69-aa58-bd312c3a1453\" (UID: \"ade89cc5-285b-4b69-aa58-bd312c3a1453\") "
Sep 10 20:06:28.494378 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:28.494230    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trlws\" (UniqueName: \"kubernetes.io/projected/ade89cc5-285b-4b69-aa58-bd312c3a1453-kube-api-access-trlws\") pod \"ade89cc5-285b-4b69-aa58-bd312c3a1453\" (UID: \"ade89cc5-285b-4b69-aa58-bd312c3a1453\") "
Sep 10 20:06:28.496396 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:28.496369    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade89cc5-285b-4b69-aa58-bd312c3a1453-kube-api-access-trlws" (OuterVolumeSpecName: "kube-api-access-trlws") pod "ade89cc5-285b-4b69-aa58-bd312c3a1453" (UID: "ade89cc5-285b-4b69-aa58-bd312c3a1453"). InnerVolumeSpecName "kube-api-access-trlws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:06:28.504265 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:28.504243    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade89cc5-285b-4b69-aa58-bd312c3a1453-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ade89cc5-285b-4b69-aa58-bd312c3a1453" (UID: "ade89cc5-285b-4b69-aa58-bd312c3a1453"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:06:28.595436 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:28.595405    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trlws\" (UniqueName: \"kubernetes.io/projected/ade89cc5-285b-4b69-aa58-bd312c3a1453-kube-api-access-trlws\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:06:28.595436 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:28.595432    2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ade89cc5-285b-4b69-aa58-bd312c3a1453-tls-cert\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:06:28.705092 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:28.705052    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:28.795347 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:28.795314    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:29.058573 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:29.058451    2561 generic.go:356] "Generic (PLEG): container finished" podID="ade89cc5-285b-4b69-aa58-bd312c3a1453" containerID="a6792ece76efbacf6b338d70492bcf0af709bf8c6d99b1340f8942c982701ba8" exitCode=0
Sep 10 20:06:29.058573 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:29.058493    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4c6c779b-x5sz6"
Sep 10 20:06:29.058573 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:29.058516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c4c6c779b-x5sz6" event={"ID":"ade89cc5-285b-4b69-aa58-bd312c3a1453","Type":"ContainerDied","Data":"a6792ece76efbacf6b338d70492bcf0af709bf8c6d99b1340f8942c982701ba8"}
Sep 10 20:06:29.058825 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:29.058588    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c4c6c779b-x5sz6" event={"ID":"ade89cc5-285b-4b69-aa58-bd312c3a1453","Type":"ContainerDied","Data":"006455e041ce1ac2c53451184113536142c4f3ea210b8b22fcf4490541cbafc9"}
Sep 10 20:06:29.058825 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:29.058608    2561 scope.go:117] "RemoveContainer" containerID="a6792ece76efbacf6b338d70492bcf0af709bf8c6d99b1340f8942c982701ba8"
Sep 10 20:06:29.068004 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:29.067982    2561 scope.go:117] "RemoveContainer" containerID="a6792ece76efbacf6b338d70492bcf0af709bf8c6d99b1340f8942c982701ba8"
Sep 10 20:06:29.068308 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:06:29.068286    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6792ece76efbacf6b338d70492bcf0af709bf8c6d99b1340f8942c982701ba8\": container with ID starting with a6792ece76efbacf6b338d70492bcf0af709bf8c6d99b1340f8942c982701ba8 not found: ID does not exist" containerID="a6792ece76efbacf6b338d70492bcf0af709bf8c6d99b1340f8942c982701ba8"
Sep 10 20:06:29.068357 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:29.068316    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6792ece76efbacf6b338d70492bcf0af709bf8c6d99b1340f8942c982701ba8"} err="failed to get container status \"a6792ece76efbacf6b338d70492bcf0af709bf8c6d99b1340f8942c982701ba8\": rpc error: code = NotFound desc = could not find container \"a6792ece76efbacf6b338d70492bcf0af709bf8c6d99b1340f8942c982701ba8\": container with ID starting with a6792ece76efbacf6b338d70492bcf0af709bf8c6d99b1340f8942c982701ba8 not found: ID does not exist"
Sep 10 20:06:29.082138 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:29.082109    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c4c6c779b-x5sz6"]
Sep 10 20:06:29.089406 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:29.089383    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c4c6c779b-x5sz6"]
Sep 10 20:06:29.403957 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:29.403882    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade89cc5-285b-4b69-aa58-bd312c3a1453" path="/var/lib/kubelet/pods/ade89cc5-285b-4b69-aa58-bd312c3a1453/volumes"
Sep 10 20:06:29.877754 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:29.877723    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:29.975781 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:29.975750    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:45.624704 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:45.624666    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:45.751928 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:45.751888    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:46.052599 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:46.052567    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:46.618586 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:46.618554    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:47.169472 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:47.169442    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:47.269722 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:47.269685    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:47.645687 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:47.645654    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:47.693049 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:47.693006    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7760-a0bc1c"]
Sep 10 20:06:47.693430 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:47.693412    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade89cc5-285b-4b69-aa58-bd312c3a1453" containerName="authorino"
Sep 10 20:06:47.693518 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:47.693433    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade89cc5-285b-4b69-aa58-bd312c3a1453" containerName="authorino"
Sep 10 20:06:47.693626 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:47.693515    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade89cc5-285b-4b69-aa58-bd312c3a1453" containerName="authorino"
Sep 10 20:06:47.695765 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:47.695743    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7760-a0bc1c"
Sep 10 20:06:47.699014 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:47.698805    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zcp8m\""
Sep 10 20:06:47.701187 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:47.701167    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7760-a0bc1c"]
Sep 10 20:06:47.761185 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:47.761147    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sstls\" (UniqueName: \"kubernetes.io/projected/8f898e96-8973-413d-aa29-7b4905057bbc-kube-api-access-sstls\") pod \"test-curl-7760-a0bc1c\" (UID: \"8f898e96-8973-413d-aa29-7b4905057bbc\") " pod="opendatahub/test-curl-7760-a0bc1c"
Sep 10 20:06:47.861619 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:47.861580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sstls\" (UniqueName: \"kubernetes.io/projected/8f898e96-8973-413d-aa29-7b4905057bbc-kube-api-access-sstls\") pod \"test-curl-7760-a0bc1c\" (UID: \"8f898e96-8973-413d-aa29-7b4905057bbc\") " pod="opendatahub/test-curl-7760-a0bc1c"
Sep 10 20:06:47.870204 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:47.870169    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sstls\" (UniqueName: \"kubernetes.io/projected/8f898e96-8973-413d-aa29-7b4905057bbc-kube-api-access-sstls\") pod \"test-curl-7760-a0bc1c\" (UID: \"8f898e96-8973-413d-aa29-7b4905057bbc\") " pod="opendatahub/test-curl-7760-a0bc1c"
Sep 10 20:06:48.007128 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:48.007033    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7760-a0bc1c"
Sep 10 20:06:48.278555 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:48.278503    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7760-a0bc1c"]
Sep 10 20:06:48.280601 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:06:48.280572    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f898e96_8973_413d_aa29_7b4905057bbc.slice/crio-e63b39049344078e7d8ce25e25732f3e95a01d5e6f5ecc2af8215ebe74a8bcd3 WatchSource:0}: Error finding container e63b39049344078e7d8ce25e25732f3e95a01d5e6f5ecc2af8215ebe74a8bcd3: Status 404 returned error can't find the container with id e63b39049344078e7d8ce25e25732f3e95a01d5e6f5ecc2af8215ebe74a8bcd3
Sep 10 20:06:49.180708 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:49.180608    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7760-a0bc1c" event={"ID":"8f898e96-8973-413d-aa29-7b4905057bbc","Type":"ContainerStarted","Data":"15c68c869c3e6c15d07fe95f2cd03cfd81ec4a390e3db97bf66085184939c717"}
Sep 10 20:06:49.180708 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:49.180656    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7760-a0bc1c" event={"ID":"8f898e96-8973-413d-aa29-7b4905057bbc","Type":"ContainerStarted","Data":"e63b39049344078e7d8ce25e25732f3e95a01d5e6f5ecc2af8215ebe74a8bcd3"}
Sep 10 20:06:49.197962 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:49.197912    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7760-a0bc1c" podStartSLOduration=1.600445514 podStartE2EDuration="2.197894009s" totalImagesPullingTime="597.448495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:06:49.196888756 +0000 UTC m=+750.247416744" watchObservedRunningTime="2026-09-10 20:06:49.197894009 +0000 UTC m=+750.248421997"
Sep 10 20:06:49.463866 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:49.463777    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:49.566861 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:49.566828    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:49.800435 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:49.800368    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7760-a0bc1c"]
Sep 10 20:06:49.814789 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:49.814750    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7760-a0bc1c"]
Sep 10 20:06:49.938451 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:49.938418    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7760-bc104b"]
Sep 10 20:06:49.938772 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:49.938759    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f898e96-8973-413d-aa29-7b4905057bbc" containerName="test-curl-7760-a0bc1c"
Sep 10 20:06:49.938895 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:49.938773    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f898e96-8973-413d-aa29-7b4905057bbc" containerName="test-curl-7760-a0bc1c"
Sep 10 20:06:49.938895 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:49.938838    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f898e96-8973-413d-aa29-7b4905057bbc" containerName="test-curl-7760-a0bc1c"
Sep 10 20:06:49.940829 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:49.940813    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7760-bc104b"
Sep 10 20:06:49.951116 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:49.951091    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7760-bc104b"]
Sep 10 20:06:49.980552 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:49.980506    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrk5\" (UniqueName: \"kubernetes.io/projected/22229911-5d91-407a-991f-0bdab2164151-kube-api-access-zvrk5\") pod \"test-curl-7760-bc104b\" (UID: \"22229911-5d91-407a-991f-0bdab2164151\") " pod="opendatahub/test-curl-7760-bc104b"
Sep 10 20:06:50.081621 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.081499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrk5\" (UniqueName: \"kubernetes.io/projected/22229911-5d91-407a-991f-0bdab2164151-kube-api-access-zvrk5\") pod \"test-curl-7760-bc104b\" (UID: \"22229911-5d91-407a-991f-0bdab2164151\") " pod="opendatahub/test-curl-7760-bc104b"
Sep 10 20:06:50.089326 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.089293    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrk5\" (UniqueName: \"kubernetes.io/projected/22229911-5d91-407a-991f-0bdab2164151-kube-api-access-zvrk5\") pod \"test-curl-7760-bc104b\" (UID: \"22229911-5d91-407a-991f-0bdab2164151\") " pod="opendatahub/test-curl-7760-bc104b"
Sep 10 20:06:50.185868 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:06:50.185835    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7760-a0bc1c" container="test-curl-7760-a0bc1c"
Sep 10 20:06:50.250319 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.250281    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7760-bc104b"
Sep 10 20:06:50.522828 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.522801    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7760-bc104b"]
Sep 10 20:06:50.524196 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:06:50.524167    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22229911_5d91_407a_991f_0bdab2164151.slice/crio-b57c421c03f6cb3d97c85f5335fa71ec7e68e44e718bc49cf17c67d0983e5698 WatchSource:0}: Error finding container b57c421c03f6cb3d97c85f5335fa71ec7e68e44e718bc49cf17c67d0983e5698: Status 404 returned error can't find the container with id b57c421c03f6cb3d97c85f5335fa71ec7e68e44e718bc49cf17c67d0983e5698
Sep 10 20:06:50.613144 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.613114    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"]
Sep 10 20:06:50.615714 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.615697    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:50.618071 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.618052    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6d8d62-serving-cert\""
Sep 10 20:06:50.626569 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.626543    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"]
Sep 10 20:06:50.658557 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.658498    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:50.687705 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.687669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cl7k\" (UniqueName: \"kubernetes.io/projected/051f910f-86bf-49fa-93ed-9189bdc21f17-kube-api-access-2cl7k\") pod \"maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:50.687913 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.687717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-maas-api-tls\") pod \"maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:50.687913 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.687741    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-metrics-certs\") pod \"maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:50.687913 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.687860    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f910f-86bf-49fa-93ed-9189bdc21f17-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:50.757654 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.757617    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:50.788824 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.788731    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f910f-86bf-49fa-93ed-9189bdc21f17-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:50.788824 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.788788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cl7k\" (UniqueName: \"kubernetes.io/projected/051f910f-86bf-49fa-93ed-9189bdc21f17-kube-api-access-2cl7k\") pod \"maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:50.789023 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.788827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-maas-api-tls\") pod \"maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:50.789023 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.788850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-metrics-certs\") pod \"maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:50.789023 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:06:50.788978    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-serving-cert: secret "maas-api-e2e-inf-a-6d8d62-serving-cert" not found
Sep 10 20:06:50.789132 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:06:50.789057    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-maas-api-tls podName:051f910f-86bf-49fa-93ed-9189bdc21f17 nodeName:}" failed. No retries permitted until 2026-09-10 20:06:51.289039593 +0000 UTC m=+752.339567560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-maas-api-tls") pod "maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl" (UID: "051f910f-86bf-49fa-93ed-9189bdc21f17") : secret "maas-api-e2e-inf-a-6d8d62-serving-cert" not found
Sep 10 20:06:50.789380 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.789361    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f910f-86bf-49fa-93ed-9189bdc21f17-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:50.791478 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.791461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-metrics-certs\") pod \"maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:50.798425 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:50.798398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cl7k\" (UniqueName: \"kubernetes.io/projected/051f910f-86bf-49fa-93ed-9189bdc21f17-kube-api-access-2cl7k\") pod \"maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:51.037715 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:51.037673    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:51.193272 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:51.193173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7760-bc104b" event={"ID":"22229911-5d91-407a-991f-0bdab2164151","Type":"ContainerStarted","Data":"239cad69aca19da607880c6ba30b64077e4ecf8eea49d5eab93abde54020b3e2"}
Sep 10 20:06:51.193272 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:51.193219    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7760-bc104b" event={"ID":"22229911-5d91-407a-991f-0bdab2164151","Type":"ContainerStarted","Data":"b57c421c03f6cb3d97c85f5335fa71ec7e68e44e718bc49cf17c67d0983e5698"}
Sep 10 20:06:51.193690 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:51.193318    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7760-a0bc1c" podUID="8f898e96-8973-413d-aa29-7b4905057bbc" containerName="test-curl-7760-a0bc1c" containerID="cri-o://15c68c869c3e6c15d07fe95f2cd03cfd81ec4a390e3db97bf66085184939c717" gracePeriod=2
Sep 10 20:06:51.203677 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:51.203652    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:51.206571 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:51.206497    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7760-bc104b" podStartSLOduration=2.206478084 podStartE2EDuration="2.206478084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:06:51.206345725 +0000 UTC m=+752.256873715" watchObservedRunningTime="2026-09-10 20:06:51.206478084 +0000 UTC m=+752.257006079"
Sep 10 20:06:51.208193 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:06:51.208167    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7760-a0bc1c\" is forbidden: User \"system:node:ip-10-0-140-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-111.ec2.internal' and this object" podUID="8f898e96-8973-413d-aa29-7b4905057bbc" pod="opendatahub/test-curl-7760-a0bc1c"
Sep 10 20:06:51.293589 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:51.293556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-maas-api-tls\") pod \"maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:51.296239 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:51.296208    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-maas-api-tls\") pod \"maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:51.300971 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:51.300945    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:51.528890 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:51.528857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:51.802974 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:51.802946    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"]
Sep 10 20:06:51.804773 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:06:51.804743    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051f910f_86bf_49fa_93ed_9189bdc21f17.slice/crio-7644764f8e1a397cd05f802ff341717155d7991583fc097137e876ec211f230a WatchSource:0}: Error finding container 7644764f8e1a397cd05f802ff341717155d7991583fc097137e876ec211f230a: Status 404 returned error can't find the container with id 7644764f8e1a397cd05f802ff341717155d7991583fc097137e876ec211f230a
Sep 10 20:06:51.819070 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:51.819033    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7760-bc104b"]
Sep 10 20:06:51.822613 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:51.822580    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7760-bc104b"]
Sep 10 20:06:52.199890 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:52.199797    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl" event={"ID":"051f910f-86bf-49fa-93ed-9189bdc21f17","Type":"ContainerStarted","Data":"7644764f8e1a397cd05f802ff341717155d7991583fc097137e876ec211f230a"}
Sep 10 20:06:52.200034 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:06:52.200023    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7760-bc104b" container="test-curl-7760-bc104b"
Sep 10 20:06:52.358288 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:52.358260    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:52.685549 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:52.685507    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:52.743482 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:52.743437    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:53.208043 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:53.208012    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl" event={"ID":"051f910f-86bf-49fa-93ed-9189bdc21f17","Type":"ContainerStarted","Data":"b4cd53c17463f61bae7aa36972eb840b40e9513e58d0214b7c89721193579ae3"}
Sep 10 20:06:53.208180 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:53.208048    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7760-bc104b" podUID="22229911-5d91-407a-991f-0bdab2164151" containerName="test-curl-7760-bc104b" containerID="cri-o://239cad69aca19da607880c6ba30b64077e4ecf8eea49d5eab93abde54020b3e2" gracePeriod=2
Sep 10 20:06:53.210200 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:06:53.210172    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7760-bc104b\" is forbidden: User \"system:node:ip-10-0-140-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-111.ec2.internal' and this object" podUID="22229911-5d91-407a-991f-0bdab2164151" pod="opendatahub/test-curl-7760-bc104b"
Sep 10 20:06:53.212110 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:06:53.212084    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7760-bc104b\" is forbidden: User \"system:node:ip-10-0-140-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-111.ec2.internal' and this object" podUID="22229911-5d91-407a-991f-0bdab2164151" pod="opendatahub/test-curl-7760-bc104b"
Sep 10 20:06:53.229879 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:53.229827    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl" podStartSLOduration=2.78333201 podStartE2EDuration="3.229811539s" totalImagesPullingTime="446.479529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:06:53.228837492 +0000 UTC m=+754.279365493" watchObservedRunningTime="2026-09-10 20:06:53.229811539 +0000 UTC m=+754.280339528"
Sep 10 20:06:53.381769 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:53.381746    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7760-a0bc1c"
Sep 10 20:06:53.383913 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:06:53.383887    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7760-bc104b\" is forbidden: User \"system:node:ip-10-0-140-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-111.ec2.internal' and this object" podUID="22229911-5d91-407a-991f-0bdab2164151" pod="opendatahub/test-curl-7760-bc104b"
Sep 10 20:06:53.514456 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:53.514360    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sstls\" (UniqueName: \"kubernetes.io/projected/8f898e96-8973-413d-aa29-7b4905057bbc-kube-api-access-sstls\") pod \"8f898e96-8973-413d-aa29-7b4905057bbc\" (UID: \"8f898e96-8973-413d-aa29-7b4905057bbc\") "
Sep 10 20:06:53.516831 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:53.516802    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f898e96-8973-413d-aa29-7b4905057bbc-kube-api-access-sstls" (OuterVolumeSpecName: "kube-api-access-sstls") pod "8f898e96-8973-413d-aa29-7b4905057bbc" (UID: "8f898e96-8973-413d-aa29-7b4905057bbc"). InnerVolumeSpecName "kube-api-access-sstls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:06:53.615300 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:53.615263    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sstls\" (UniqueName: \"kubernetes.io/projected/8f898e96-8973-413d-aa29-7b4905057bbc-kube-api-access-sstls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:06:53.831860 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:53.831828    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:53.932578 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:53.932540    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:54.074730 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:54.074697    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:54.214908 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:54.214806    2561 generic.go:356] "Generic (PLEG): container finished" podID="8f898e96-8973-413d-aa29-7b4905057bbc" containerID="15c68c869c3e6c15d07fe95f2cd03cfd81ec4a390e3db97bf66085184939c717" exitCode=137
Sep 10 20:06:54.214908 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:54.214860    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7760-a0bc1c"
Sep 10 20:06:54.215146 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:54.214915    2561 scope.go:117] "RemoveContainer" containerID="15c68c869c3e6c15d07fe95f2cd03cfd81ec4a390e3db97bf66085184939c717"
Sep 10 20:06:54.215146 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:54.215080    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:06:54.224370 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:54.224353    2561 scope.go:117] "RemoveContainer" containerID="15c68c869c3e6c15d07fe95f2cd03cfd81ec4a390e3db97bf66085184939c717"
Sep 10 20:06:54.224683 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:06:54.224667    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c68c869c3e6c15d07fe95f2cd03cfd81ec4a390e3db97bf66085184939c717\": container with ID starting with 15c68c869c3e6c15d07fe95f2cd03cfd81ec4a390e3db97bf66085184939c717 not found: ID does not exist" containerID="15c68c869c3e6c15d07fe95f2cd03cfd81ec4a390e3db97bf66085184939c717"
Sep 10 20:06:54.224754 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:54.224691    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c68c869c3e6c15d07fe95f2cd03cfd81ec4a390e3db97bf66085184939c717"} err="failed to get container status \"15c68c869c3e6c15d07fe95f2cd03cfd81ec4a390e3db97bf66085184939c717\": rpc error: code = NotFound desc = could not find container \"15c68c869c3e6c15d07fe95f2cd03cfd81ec4a390e3db97bf66085184939c717\": container with ID starting with 15c68c869c3e6c15d07fe95f2cd03cfd81ec4a390e3db97bf66085184939c717 not found: ID does not exist"
Sep 10 20:06:54.408165 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:54.408132    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:54.770124 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:54.770088    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:55.363881 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:55.363849    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:55.395635 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:55.395609    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7760-bc104b"
Sep 10 20:06:55.403692 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:55.403664    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f898e96-8973-413d-aa29-7b4905057bbc" path="/var/lib/kubelet/pods/8f898e96-8973-413d-aa29-7b4905057bbc/volumes"
Sep 10 20:06:55.465408 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:55.465327    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:55.532103 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:55.532073    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvrk5\" (UniqueName: \"kubernetes.io/projected/22229911-5d91-407a-991f-0bdab2164151-kube-api-access-zvrk5\") pod \"22229911-5d91-407a-991f-0bdab2164151\" (UID: \"22229911-5d91-407a-991f-0bdab2164151\") "
Sep 10 20:06:55.534415 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:55.534379    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22229911-5d91-407a-991f-0bdab2164151-kube-api-access-zvrk5" (OuterVolumeSpecName: "kube-api-access-zvrk5") pod "22229911-5d91-407a-991f-0bdab2164151" (UID: "22229911-5d91-407a-991f-0bdab2164151"). InnerVolumeSpecName "kube-api-access-zvrk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:06:55.633714 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:55.633677    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvrk5\" (UniqueName: \"kubernetes.io/projected/22229911-5d91-407a-991f-0bdab2164151-kube-api-access-zvrk5\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:06:56.228397 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.228363    2561 generic.go:356] "Generic (PLEG): container finished" podID="22229911-5d91-407a-991f-0bdab2164151" containerID="239cad69aca19da607880c6ba30b64077e4ecf8eea49d5eab93abde54020b3e2" exitCode=137
Sep 10 20:06:56.228612 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.228408    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7760-bc104b"
Sep 10 20:06:56.228612 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.228464    2561 scope.go:117] "RemoveContainer" containerID="239cad69aca19da607880c6ba30b64077e4ecf8eea49d5eab93abde54020b3e2"
Sep 10 20:06:56.238024 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.238007    2561 scope.go:117] "RemoveContainer" containerID="239cad69aca19da607880c6ba30b64077e4ecf8eea49d5eab93abde54020b3e2"
Sep 10 20:06:56.238358 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:06:56.238339    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239cad69aca19da607880c6ba30b64077e4ecf8eea49d5eab93abde54020b3e2\": container with ID starting with 239cad69aca19da607880c6ba30b64077e4ecf8eea49d5eab93abde54020b3e2 not found: ID does not exist" containerID="239cad69aca19da607880c6ba30b64077e4ecf8eea49d5eab93abde54020b3e2"
Sep 10 20:06:56.238419 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.238369    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239cad69aca19da607880c6ba30b64077e4ecf8eea49d5eab93abde54020b3e2"} err="failed to get container status \"239cad69aca19da607880c6ba30b64077e4ecf8eea49d5eab93abde54020b3e2\": rpc error: code = NotFound desc = could not find container \"239cad69aca19da607880c6ba30b64077e4ecf8eea49d5eab93abde54020b3e2\": container with ID starting with 239cad69aca19da607880c6ba30b64077e4ecf8eea49d5eab93abde54020b3e2 not found: ID does not exist"
Sep 10 20:06:56.495877 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.495796    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"]
Sep 10 20:06:56.496217 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.496143    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22229911-5d91-407a-991f-0bdab2164151" containerName="test-curl-7760-bc104b"
Sep 10 20:06:56.496217 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.496154    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="22229911-5d91-407a-991f-0bdab2164151" containerName="test-curl-7760-bc104b"
Sep 10 20:06:56.496217 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.496217    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="22229911-5d91-407a-991f-0bdab2164151" containerName="test-curl-7760-bc104b"
Sep 10 20:06:56.498828 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.498809    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:56.501423 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.501401    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-28a5f0-serving-cert\""
Sep 10 20:06:56.511504 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.511479    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"]
Sep 10 20:06:56.549960 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.549926    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:56.641660 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.641620    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dd2460e-6774-4345-9da6-4b85ea993154-openshift-service-ca\") pod \"maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:56.641660 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.641661    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dd2460e-6774-4345-9da6-4b85ea993154-metrics-certs\") pod \"maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:56.641881 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.641750    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chnnb\" (UniqueName: \"kubernetes.io/projected/3dd2460e-6774-4345-9da6-4b85ea993154-kube-api-access-chnnb\") pod \"maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:56.641881 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.641785    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dd2460e-6774-4345-9da6-4b85ea993154-maas-api-tls\") pod \"maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:56.650457 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.650428    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:56.742992 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.742957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chnnb\" (UniqueName: \"kubernetes.io/projected/3dd2460e-6774-4345-9da6-4b85ea993154-kube-api-access-chnnb\") pod \"maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:56.743210 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.743012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dd2460e-6774-4345-9da6-4b85ea993154-maas-api-tls\") pod \"maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:56.743210 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.743088    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dd2460e-6774-4345-9da6-4b85ea993154-openshift-service-ca\") pod \"maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:56.743210 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.743110    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dd2460e-6774-4345-9da6-4b85ea993154-metrics-certs\") pod \"maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:56.743802 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.743767    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dd2460e-6774-4345-9da6-4b85ea993154-openshift-service-ca\") pod \"maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:56.745766 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.745739    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dd2460e-6774-4345-9da6-4b85ea993154-maas-api-tls\") pod \"maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:56.745852 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.745789    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dd2460e-6774-4345-9da6-4b85ea993154-metrics-certs\") pod \"maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:56.755752 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.755650    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:56.758475 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.758455    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chnnb\" (UniqueName: \"kubernetes.io/projected/3dd2460e-6774-4345-9da6-4b85ea993154-kube-api-access-chnnb\") pod \"maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:56.811934 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.811895    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:56.847936 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:56.847899    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:57.111181 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:57.111158    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"]
Sep 10 20:06:57.112572 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:06:57.112546    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd2460e_6774_4345_9da6_4b85ea993154.slice/crio-d5a50b05e10889e73ce6825b2cfac97fad441f9f5b473e098329df397a8682c0 WatchSource:0}: Error finding container d5a50b05e10889e73ce6825b2cfac97fad441f9f5b473e098329df397a8682c0: Status 404 returned error can't find the container with id d5a50b05e10889e73ce6825b2cfac97fad441f9f5b473e098329df397a8682c0
Sep 10 20:06:57.234893 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:57.234860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn" event={"ID":"3dd2460e-6774-4345-9da6-4b85ea993154","Type":"ContainerStarted","Data":"d5a50b05e10889e73ce6825b2cfac97fad441f9f5b473e098329df397a8682c0"}
Sep 10 20:06:57.251282 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:57.251250    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:57.346244 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:57.346208    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:57.410982 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:57.410922    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22229911-5d91-407a-991f-0bdab2164151" path="/var/lib/kubelet/pods/22229911-5d91-407a-991f-0bdab2164151/volumes"
Sep 10 20:06:58.244084 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.244041    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn" event={"ID":"3dd2460e-6774-4345-9da6-4b85ea993154","Type":"ContainerStarted","Data":"8378a3b4549777a7fb5406a10066374f4fa3734889970f611b06dcd0ec0440af"}
Sep 10 20:06:58.244556 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.244222    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:06:58.264843 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.264786    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn" podStartSLOduration=1.88648545 podStartE2EDuration="2.264767304s" totalImagesPullingTime="378.281854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:06:58.262200218 +0000 UTC m=+759.312728209" watchObservedRunningTime="2026-09-10 20:06:58.264767304 +0000 UTC m=+759.315295293"
Sep 10 20:06:58.436296 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.436261    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"]
Sep 10 20:06:58.439338 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.439312    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:58.441948 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.441923    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cde8cba8-serving-cert\""
Sep 10 20:06:58.453204 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.453144    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"]
Sep 10 20:06:58.482671 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.482637    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:58.560965 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.560925    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-metrics-certs\") pod \"maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:58.561148 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.560969    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp8mj\" (UniqueName: \"kubernetes.io/projected/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-kube-api-access-xp8mj\") pod \"maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:58.561148 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.561031    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-maas-api-tls\") pod \"maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:58.561148 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.561102    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-openshift-service-ca\") pod \"maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:58.581465 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.581434    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:58.637020 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.636982    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"]
Sep 10 20:06:58.640005 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.639986    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.642147 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.642128    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-6e5a8205-kserve-self-signed-certs\""
Sep 10 20:06:58.647505 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.647372    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"]
Sep 10 20:06:58.661733 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.661696    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-openshift-service-ca\") pod \"maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:58.661920 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.661807    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-metrics-certs\") pod \"maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:58.661920 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.661842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8mj\" (UniqueName: \"kubernetes.io/projected/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-kube-api-access-xp8mj\") pod \"maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:58.661920 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.661869    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-maas-api-tls\") pod \"maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:58.662103 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:06:58.661995    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-serving-cert: secret "maas-api-e2e-ait-cde8cba8-serving-cert" not found
Sep 10 20:06:58.662103 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:06:58.662096    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-maas-api-tls podName:80c2f1c1-a486-4cd0-9b14-bd3f0c231703 nodeName:}" failed. No retries permitted until 2026-09-10 20:06:59.162075036 +0000 UTC m=+760.212603021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-maas-api-tls") pod "maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr" (UID: "80c2f1c1-a486-4cd0-9b14-bd3f0c231703") : secret "maas-api-e2e-ait-cde8cba8-serving-cert" not found
Sep 10 20:06:58.662483 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.662460    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-openshift-service-ca\") pod \"maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:58.664589 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.664570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-metrics-certs\") pod \"maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:58.672127 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.672095    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8mj\" (UniqueName: \"kubernetes.io/projected/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-kube-api-access-xp8mj\") pod \"maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:58.762791 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.762748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-model-cache\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.762791 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.762789    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-dshm\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.763016 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.762812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpw5r\" (UniqueName: \"kubernetes.io/projected/aedbd723-f28d-4777-af21-996699b08ec1-kube-api-access-vpw5r\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.763016 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.762866    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aedbd723-f28d-4777-af21-996699b08ec1-tls-certs\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.763016 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.762931    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-home\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.763016 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.762971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-tmp-dir\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.789722 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.789685    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:58.864182 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.864079    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-model-cache\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.864182 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.864122    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-dshm\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.864182 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.864146    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpw5r\" (UniqueName: \"kubernetes.io/projected/aedbd723-f28d-4777-af21-996699b08ec1-kube-api-access-vpw5r\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.864182 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.864173    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aedbd723-f28d-4777-af21-996699b08ec1-tls-certs\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.864555 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.864238    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-home\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.864555 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.864264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-tmp-dir\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.864669 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.864596    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-model-cache\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.864669 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.864621    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-home\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.864766 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.864679    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-tmp-dir\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.866588 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.866567    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-dshm\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.866760 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.866743    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aedbd723-f28d-4777-af21-996699b08ec1-tls-certs\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.874597 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.874553    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpw5r\" (UniqueName: \"kubernetes.io/projected/aedbd723-f28d-4777-af21-996699b08ec1-kube-api-access-vpw5r\") pod \"e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") " pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.950232 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.950192    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:06:58.967994 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:58.967958    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:59.095547 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:59.095490    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:59.167848 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:59.167757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-maas-api-tls\") pod \"maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:59.170249 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:59.170219    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-maas-api-tls\") pod \"maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:59.195990 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:59.195955    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:06:59.237816 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:59.237785    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"]
Sep 10 20:06:59.238839 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:06:59.238815    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaedbd723_f28d_4777_af21_996699b08ec1.slice/crio-1b35c3969470c02ca0bea9dd80447ba918cb901c21a17ff94e3ad29c37c4ae63 WatchSource:0}: Error finding container 1b35c3969470c02ca0bea9dd80447ba918cb901c21a17ff94e3ad29c37c4ae63: Status 404 returned error can't find the container with id 1b35c3969470c02ca0bea9dd80447ba918cb901c21a17ff94e3ad29c37c4ae63
Sep 10 20:06:59.252512 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:59.252473    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml" event={"ID":"aedbd723-f28d-4777-af21-996699b08ec1","Type":"ContainerStarted","Data":"1b35c3969470c02ca0bea9dd80447ba918cb901c21a17ff94e3ad29c37c4ae63"}
Sep 10 20:06:59.352295 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:59.352260    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:06:59.629967 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:06:59.629940    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"]
Sep 10 20:06:59.631200 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:06:59.631170    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c2f1c1_a486_4cd0_9b14_bd3f0c231703.slice/crio-89fec434da86cd6f1e319f7c0e0adad3be147adf4be9ae1fff4acbb1ee106fd9 WatchSource:0}: Error finding container 89fec434da86cd6f1e319f7c0e0adad3be147adf4be9ae1fff4acbb1ee106fd9: Status 404 returned error can't find the container with id 89fec434da86cd6f1e319f7c0e0adad3be147adf4be9ae1fff4acbb1ee106fd9
Sep 10 20:07:00.227361 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:00.227320    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:07:00.262322 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:00.262222    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr" event={"ID":"80c2f1c1-a486-4cd0-9b14-bd3f0c231703","Type":"ContainerStarted","Data":"cb1b33930754ebd3948c9e4438795a2ac0e9b5dd535b748024524f7a4570a46e"}
Sep 10 20:07:00.262322 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:00.262282    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr" event={"ID":"80c2f1c1-a486-4cd0-9b14-bd3f0c231703","Type":"ContainerStarted","Data":"89fec434da86cd6f1e319f7c0e0adad3be147adf4be9ae1fff4acbb1ee106fd9"}
Sep 10 20:07:00.262866 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:00.262414    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:07:00.264265 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:00.264232    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml" event={"ID":"aedbd723-f28d-4777-af21-996699b08ec1","Type":"ContainerStarted","Data":"58bf4708e895c6af3d9d328e931817a94a7a81e98f58f977b98829477826dad5"}
Sep 10 20:07:00.284772 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:00.284708    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr" podStartSLOduration=1.917877377 podStartE2EDuration="2.284685743s" totalImagesPullingTime="366.808366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:07:00.280953942 +0000 UTC m=+761.331481933" watchObservedRunningTime="2026-09-10 20:07:00.284685743 +0000 UTC m=+761.335213735"
Sep 10 20:07:00.298517 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:00.298462    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml" podStartSLOduration=2.298445675 podStartE2EDuration="2.298445675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:07:00.296209293 +0000 UTC m=+761.346737282" watchObservedRunningTime="2026-09-10 20:07:00.298445675 +0000 UTC m=+761.348973687"
Sep 10 20:07:04.258585 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:04.258550    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:07:04.889077 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:04.889043    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:05.056922 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:05.056882    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:05.201725 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:05.201640    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:05.449584 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:05.449522    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:06.275130 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:06.275100    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:07:07.984137 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:07.984099    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:08.950927 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:08.950888    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:07:08.951113 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:08.950947    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:07:08.963476 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:08.963448    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:07:09.323777 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:09.323750    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:07:09.757508 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:09.757408    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:11.835517 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:11.835484    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:14.358436 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:14.358395    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:14.701296 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:14.701189    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:15.025283 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:15.025251    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:15.434940 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:15.434865    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:17.507699 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:17.507669    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:17.602716 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:17.602687    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:17.752220 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:17.752183    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:17.852898 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:17.852865    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:18.727639 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:18.727603    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:18.824414 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:18.824384    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:19.954228 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:19.954189    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:20.359386 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:20.359353    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:20.916403 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:20.916364    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"]
Sep 10 20:07:20.921674 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:20.921648    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:20.924417 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:20.924394    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-39487c-serving-cert\""
Sep 10 20:07:20.931562 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:20.931520    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"]
Sep 10 20:07:20.954624 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:20.954595    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:20.980665 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:20.980635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0151370b-d749-4880-8c7b-9a50be65a2a6-metrics-certs\") pod \"maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:20.980823 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:20.980677    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0151370b-d749-4880-8c7b-9a50be65a2a6-maas-api-tls\") pod \"maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:20.980823 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:20.980698    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0151370b-d749-4880-8c7b-9a50be65a2a6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:20.980891 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:20.980829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68bjb\" (UniqueName: \"kubernetes.io/projected/0151370b-d749-4880-8c7b-9a50be65a2a6-kube-api-access-68bjb\") pod \"maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:21.024693 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.024664    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:21.082225 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.082189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68bjb\" (UniqueName: \"kubernetes.io/projected/0151370b-d749-4880-8c7b-9a50be65a2a6-kube-api-access-68bjb\") pod \"maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:21.082448 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.082248    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0151370b-d749-4880-8c7b-9a50be65a2a6-metrics-certs\") pod \"maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:21.082448 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.082419    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0151370b-d749-4880-8c7b-9a50be65a2a6-maas-api-tls\") pod \"maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:21.082559 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.082454    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0151370b-d749-4880-8c7b-9a50be65a2a6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:21.083186 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.083169    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0151370b-d749-4880-8c7b-9a50be65a2a6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:21.084843 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.084818    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0151370b-d749-4880-8c7b-9a50be65a2a6-metrics-certs\") pod \"maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:21.085084 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.085064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0151370b-d749-4880-8c7b-9a50be65a2a6-maas-api-tls\") pod \"maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:21.091378 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.091354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68bjb\" (UniqueName: \"kubernetes.io/projected/0151370b-d749-4880-8c7b-9a50be65a2a6-kube-api-access-68bjb\") pod \"maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:21.125751 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.125721    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:21.236137 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.236043    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:21.468746 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.468711    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:21.514870 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.514847    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"]
Sep 10 20:07:21.515987 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:07:21.515964    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0151370b_d749_4880_8c7b_9a50be65a2a6.slice/crio-11e8f3b0d2fb44754b2887a908bf99bec2160377e41786e34c963289ad1c315f WatchSource:0}: Error finding container 11e8f3b0d2fb44754b2887a908bf99bec2160377e41786e34c963289ad1c315f: Status 404 returned error can't find the container with id 11e8f3b0d2fb44754b2887a908bf99bec2160377e41786e34c963289ad1c315f
Sep 10 20:07:21.547873 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.547828    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:21.870936 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:21.870902    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:22.382776 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:22.382745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q" event={"ID":"0151370b-d749-4880-8c7b-9a50be65a2a6","Type":"ContainerStarted","Data":"6f52f7cfe4b46d28eacfaa861af86490653ddafbda0ab973d75b0bd1416ea09e"}
Sep 10 20:07:22.383163 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:22.382827    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q" event={"ID":"0151370b-d749-4880-8c7b-9a50be65a2a6","Type":"ContainerStarted","Data":"11e8f3b0d2fb44754b2887a908bf99bec2160377e41786e34c963289ad1c315f"}
Sep 10 20:07:22.383163 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:22.382876    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:22.401115 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:22.401066    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q" podStartSLOduration=1.946397019 podStartE2EDuration="2.401050688s" totalImagesPullingTime="454.653669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:07:22.400008183 +0000 UTC m=+783.450536173" watchObservedRunningTime="2026-09-10 20:07:22.401050688 +0000 UTC m=+783.451578729"
Sep 10 20:07:22.852246 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:22.852208    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:23.732478 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:23.732450    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:23.832722 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:23.832682    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:23.941078 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:23.941046    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:24.034475 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:24.034439    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:24.229457 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:24.229426    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:28.393604 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:28.393574    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:07:30.242581 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:30.242512    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:30.347311 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:30.347278    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:38.493787 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.493741    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"]
Sep 10 20:07:38.494244 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.494019    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr" podUID="80c2f1c1-a486-4cd0-9b14-bd3f0c231703" containerName="maas-api" containerID="cri-o://cb1b33930754ebd3948c9e4438795a2ac0e9b5dd535b748024524f7a4570a46e" gracePeriod=30
Sep 10 20:07:38.517810 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.517780    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:38.618017 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.617962    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:38.712870 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.712814    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:38.742653 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.742628    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:07:38.784753 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.784725    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:38.850273 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.850237    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp8mj\" (UniqueName: \"kubernetes.io/projected/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-kube-api-access-xp8mj\") pod \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") "
Sep 10 20:07:38.850515 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.850287    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-maas-api-tls\") pod \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") "
Sep 10 20:07:38.850515 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.850316    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-metrics-certs\") pod \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") "
Sep 10 20:07:38.850515 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.850398    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-openshift-service-ca\") pod \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\" (UID: \"80c2f1c1-a486-4cd0-9b14-bd3f0c231703\") "
Sep 10 20:07:38.850942 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.850881    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "80c2f1c1-a486-4cd0-9b14-bd3f0c231703" (UID: "80c2f1c1-a486-4cd0-9b14-bd3f0c231703"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:07:38.852932 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.852907    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "80c2f1c1-a486-4cd0-9b14-bd3f0c231703" (UID: "80c2f1c1-a486-4cd0-9b14-bd3f0c231703"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:07:38.853051 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.852949    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "80c2f1c1-a486-4cd0-9b14-bd3f0c231703" (UID: "80c2f1c1-a486-4cd0-9b14-bd3f0c231703"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:07:38.853051 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.852988    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-kube-api-access-xp8mj" (OuterVolumeSpecName: "kube-api-access-xp8mj") pod "80c2f1c1-a486-4cd0-9b14-bd3f0c231703" (UID: "80c2f1c1-a486-4cd0-9b14-bd3f0c231703"). InnerVolumeSpecName "kube-api-access-xp8mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:07:38.857033 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.857007    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:38.951130 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.951095    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:07:38.951130 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.951130    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp8mj\" (UniqueName: \"kubernetes.io/projected/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-kube-api-access-xp8mj\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:07:38.951130 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.951144    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:07:38.951386 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:38.951158    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c2f1c1-a486-4cd0-9b14-bd3f0c231703-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:07:39.341813 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:39.341781    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:39.421659 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:39.421625    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:39.474941 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:39.474910    2561 generic.go:356] "Generic (PLEG): container finished" podID="80c2f1c1-a486-4cd0-9b14-bd3f0c231703" containerID="cb1b33930754ebd3948c9e4438795a2ac0e9b5dd535b748024524f7a4570a46e" exitCode=0
Sep 10 20:07:39.475154 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:39.474964    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"
Sep 10 20:07:39.475154 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:39.475000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr" event={"ID":"80c2f1c1-a486-4cd0-9b14-bd3f0c231703","Type":"ContainerDied","Data":"cb1b33930754ebd3948c9e4438795a2ac0e9b5dd535b748024524f7a4570a46e"}
Sep 10 20:07:39.475154 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:39.475054    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr" event={"ID":"80c2f1c1-a486-4cd0-9b14-bd3f0c231703","Type":"ContainerDied","Data":"89fec434da86cd6f1e319f7c0e0adad3be147adf4be9ae1fff4acbb1ee106fd9"}
Sep 10 20:07:39.475154 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:39.475078    2561 scope.go:117] "RemoveContainer" containerID="cb1b33930754ebd3948c9e4438795a2ac0e9b5dd535b748024524f7a4570a46e"
Sep 10 20:07:39.485019 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:39.485000    2561 scope.go:117] "RemoveContainer" containerID="cb1b33930754ebd3948c9e4438795a2ac0e9b5dd535b748024524f7a4570a46e"
Sep 10 20:07:39.485323 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:07:39.485304    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb1b33930754ebd3948c9e4438795a2ac0e9b5dd535b748024524f7a4570a46e\": container with ID starting with cb1b33930754ebd3948c9e4438795a2ac0e9b5dd535b748024524f7a4570a46e not found: ID does not exist" containerID="cb1b33930754ebd3948c9e4438795a2ac0e9b5dd535b748024524f7a4570a46e"
Sep 10 20:07:39.485367 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:39.485332    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb1b33930754ebd3948c9e4438795a2ac0e9b5dd535b748024524f7a4570a46e"} err="failed to get container status \"cb1b33930754ebd3948c9e4438795a2ac0e9b5dd535b748024524f7a4570a46e\": rpc error: code = NotFound desc = could not find container \"cb1b33930754ebd3948c9e4438795a2ac0e9b5dd535b748024524f7a4570a46e\": container with ID starting with cb1b33930754ebd3948c9e4438795a2ac0e9b5dd535b748024524f7a4570a46e not found: ID does not exist"
Sep 10 20:07:39.499493 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:39.499460    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"]
Sep 10 20:07:39.507832 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:39.507795    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cde8cba8-5f47fd9bd-k5ffr"]
Sep 10 20:07:39.510189 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:39.510154    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:39.794090 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:39.794061    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:40.161880 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:40.161785    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:41.405206 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:41.405169    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80c2f1c1-a486-4cd0-9b14-bd3f0c231703" path="/var/lib/kubelet/pods/80c2f1c1-a486-4cd0-9b14-bd3f0c231703/volumes"
Sep 10 20:07:42.045657 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:42.045624    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:42.146495 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:42.146459    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:42.911989 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:42.911955    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"]
Sep 10 20:07:42.912813 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:42.912789    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c2f1c1-a486-4cd0-9b14-bd3f0c231703" containerName="maas-api"
Sep 10 20:07:42.912970 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:42.912866    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c2f1c1-a486-4cd0-9b14-bd3f0c231703" containerName="maas-api"
Sep 10 20:07:42.913053 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:42.913034    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="80c2f1c1-a486-4cd0-9b14-bd3f0c231703" containerName="maas-api"
Sep 10 20:07:42.917613 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:42.917594    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:42.920029 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:42.920007    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f563295c-serving-cert\""
Sep 10 20:07:42.927630 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:42.927608    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"]
Sep 10 20:07:42.943425 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:42.943389    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:42.987782 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:42.987748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7td8\" (UniqueName: \"kubernetes.io/projected/8685b5d7-4581-4581-9c4f-14f8a7c35f69-kube-api-access-q7td8\") pod \"maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:42.987959 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:42.987798    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8685b5d7-4581-4581-9c4f-14f8a7c35f69-openshift-service-ca\") pod \"maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:42.987959 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:42.987882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-maas-api-tls\") pod \"maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:42.987959 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:42.987910    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-metrics-certs\") pod \"maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:43.004904 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.004862    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:43.089159 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.089128    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-maas-api-tls\") pod \"maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:43.089159 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.089164    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-metrics-certs\") pod \"maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:43.089397 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.089235    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7td8\" (UniqueName: \"kubernetes.io/projected/8685b5d7-4581-4581-9c4f-14f8a7c35f69-kube-api-access-q7td8\") pod \"maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:43.089397 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.089273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8685b5d7-4581-4581-9c4f-14f8a7c35f69-openshift-service-ca\") pod \"maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:43.089397 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:07:43.089286    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-serving-cert: secret "maas-api-e2e-migrate-f563295c-serving-cert" not found
Sep 10 20:07:43.089397 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:07:43.089363    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-maas-api-tls podName:8685b5d7-4581-4581-9c4f-14f8a7c35f69 nodeName:}" failed. No retries permitted until 2026-09-10 20:07:43.589344893 +0000 UTC m=+804.639872865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-maas-api-tls") pod "maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9" (UID: "8685b5d7-4581-4581-9c4f-14f8a7c35f69") : secret "maas-api-e2e-migrate-f563295c-serving-cert" not found
Sep 10 20:07:43.089951 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.089926    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8685b5d7-4581-4581-9c4f-14f8a7c35f69-openshift-service-ca\") pod \"maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:43.091845 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.091824    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-metrics-certs\") pod \"maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:43.097091 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.097065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7td8\" (UniqueName: \"kubernetes.io/projected/8685b5d7-4581-4581-9c4f-14f8a7c35f69-kube-api-access-q7td8\") pod \"maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:43.330942 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.330898    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:43.436960 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.436928    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:43.534892 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.534849    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:43.595276 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.595190    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-maas-api-tls\") pod \"maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:43.597823 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.597801    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-maas-api-tls\") pod \"maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:43.627707 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.627678    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:43.829881 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:43.829835    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:44.028017 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:44.027986    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:44.104857 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:44.104831    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"]
Sep 10 20:07:44.107542 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:07:44.107490    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8685b5d7_4581_4581_9c4f_14f8a7c35f69.slice/crio-622b455b621a570172c357fb61a6139a97e18cbda4fdc77ab2b4440f62ee36bd WatchSource:0}: Error finding container 622b455b621a570172c357fb61a6139a97e18cbda4fdc77ab2b4440f62ee36bd: Status 404 returned error can't find the container with id 622b455b621a570172c357fb61a6139a97e18cbda4fdc77ab2b4440f62ee36bd
Sep 10 20:07:44.506392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:44.506348    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9" event={"ID":"8685b5d7-4581-4581-9c4f-14f8a7c35f69","Type":"ContainerStarted","Data":"622b455b621a570172c357fb61a6139a97e18cbda4fdc77ab2b4440f62ee36bd"}
Sep 10 20:07:44.854870 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:44.854836    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:07:45.513240 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:45.513200    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9" event={"ID":"8685b5d7-4581-4581-9c4f-14f8a7c35f69","Type":"ContainerStarted","Data":"9e43cc4d83fc75e5c3eb94cc74c00f35311f25097b99ffa64bd8c04946ffb74d"}
Sep 10 20:07:45.513681 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:45.513290    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:07:45.529617 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:45.529566    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9" podStartSLOduration=3.005404314 podStartE2EDuration="3.529548943s" totalImagesPullingTime="524.144629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:07:45.527501713 +0000 UTC m=+806.578029767" watchObservedRunningTime="2026-09-10 20:07:45.529548943 +0000 UTC m=+806.580076929"
Sep 10 20:07:51.525301 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:07:51.525269    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:08:14.065220 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.065182    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"]
Sep 10 20:08:14.068522 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.068504    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.071184 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.071165    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-6e5a8205-kserve-self-signed-certs\""
Sep 10 20:08:14.078852 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.078833    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"]
Sep 10 20:08:14.150158 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.150127    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:14.170399 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.170363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-model-cache\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.170399 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.170405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174b6c1f-21c0-4b75-84fc-5765c46f5d24-tls-certs\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.170645 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.170430    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-dshm\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.170645 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.170543    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-home\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.170645 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.170605    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vlnc\" (UniqueName: \"kubernetes.io/projected/174b6c1f-21c0-4b75-84fc-5765c46f5d24-kube-api-access-5vlnc\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.170763 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.170684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-tmp-dir\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.272093 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.272050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-tmp-dir\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.272272 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.272111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-model-cache\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.272272 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.272146    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174b6c1f-21c0-4b75-84fc-5765c46f5d24-tls-certs\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.272272 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.272172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-dshm\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.272272 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.272192    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-home\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.272272 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.272229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vlnc\" (UniqueName: \"kubernetes.io/projected/174b6c1f-21c0-4b75-84fc-5765c46f5d24-kube-api-access-5vlnc\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.272592 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.272514    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-tmp-dir\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.272592 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.272564    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-model-cache\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.272724 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.272680    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-home\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.274729 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.274701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-dshm\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.275038 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.275018    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174b6c1f-21c0-4b75-84fc-5765c46f5d24-tls-certs\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.280446 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.280423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vlnc\" (UniqueName: \"kubernetes.io/projected/174b6c1f-21c0-4b75-84fc-5765c46f5d24-kube-api-access-5vlnc\") pod \"e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") " pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.378821 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.378737    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:14.653690 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.653660    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"]
Sep 10 20:08:14.655001 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:08:14.654979    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod174b6c1f_21c0_4b75_84fc_5765c46f5d24.slice/crio-963ad28df15dea6bc28724461291bb52e01aa0f45808bcebd366a8eae06b2ae1 WatchSource:0}: Error finding container 963ad28df15dea6bc28724461291bb52e01aa0f45808bcebd366a8eae06b2ae1: Status 404 returned error can't find the container with id 963ad28df15dea6bc28724461291bb52e01aa0f45808bcebd366a8eae06b2ae1
Sep 10 20:08:14.672302 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:14.672276    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n" event={"ID":"174b6c1f-21c0-4b75-84fc-5765c46f5d24","Type":"ContainerStarted","Data":"963ad28df15dea6bc28724461291bb52e01aa0f45808bcebd366a8eae06b2ae1"}
Sep 10 20:08:15.679430 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:15.679389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n" event={"ID":"174b6c1f-21c0-4b75-84fc-5765c46f5d24","Type":"ContainerStarted","Data":"1fd7de028d5ea32e60cac2e1da5fb3f211e5dfcd6f365f2ecea700ca759faa8d"}
Sep 10 20:08:15.697382 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:15.697328    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n" podStartSLOduration=1.6973118820000002 podStartE2EDuration="1.697311882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:08:15.695305071 +0000 UTC m=+836.745833066" watchObservedRunningTime="2026-09-10 20:08:15.697311882 +0000 UTC m=+836.747839871"
Sep 10 20:08:18.095859 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:18.095819    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:23.607834 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.607793    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"]
Sep 10 20:08:23.608372 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.608081    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9" podUID="8685b5d7-4581-4581-9c4f-14f8a7c35f69" containerName="maas-api" containerID="cri-o://9e43cc4d83fc75e5c3eb94cc74c00f35311f25097b99ffa64bd8c04946ffb74d" gracePeriod=30
Sep 10 20:08:23.642616 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.642582    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:23.710359 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.710328    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:23.725591 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.725556    2561 generic.go:356] "Generic (PLEG): container finished" podID="8685b5d7-4581-4581-9c4f-14f8a7c35f69" containerID="9e43cc4d83fc75e5c3eb94cc74c00f35311f25097b99ffa64bd8c04946ffb74d" exitCode=0
Sep 10 20:08:23.725722 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.725572    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9" event={"ID":"8685b5d7-4581-4581-9c4f-14f8a7c35f69","Type":"ContainerDied","Data":"9e43cc4d83fc75e5c3eb94cc74c00f35311f25097b99ffa64bd8c04946ffb74d"}
Sep 10 20:08:23.773941 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.773905    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:23.844948 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.844921    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:23.861750 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.861684    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:08:23.968283 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.968254    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-metrics-certs\") pod \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") "
Sep 10 20:08:23.968452 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.968318    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8685b5d7-4581-4581-9c4f-14f8a7c35f69-openshift-service-ca\") pod \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") "
Sep 10 20:08:23.968452 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.968388    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-maas-api-tls\") pod \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") "
Sep 10 20:08:23.968452 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.968428    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7td8\" (UniqueName: \"kubernetes.io/projected/8685b5d7-4581-4581-9c4f-14f8a7c35f69-kube-api-access-q7td8\") pod \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\" (UID: \"8685b5d7-4581-4581-9c4f-14f8a7c35f69\") "
Sep 10 20:08:23.968755 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.968726    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8685b5d7-4581-4581-9c4f-14f8a7c35f69-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8685b5d7-4581-4581-9c4f-14f8a7c35f69" (UID: "8685b5d7-4581-4581-9c4f-14f8a7c35f69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:08:23.970622 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.970595    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8685b5d7-4581-4581-9c4f-14f8a7c35f69" (UID: "8685b5d7-4581-4581-9c4f-14f8a7c35f69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:08:23.970737 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.970622    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8685b5d7-4581-4581-9c4f-14f8a7c35f69-kube-api-access-q7td8" (OuterVolumeSpecName: "kube-api-access-q7td8") pod "8685b5d7-4581-4581-9c4f-14f8a7c35f69" (UID: "8685b5d7-4581-4581-9c4f-14f8a7c35f69"). InnerVolumeSpecName "kube-api-access-q7td8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:08:23.970787 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.970749    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8685b5d7-4581-4581-9c4f-14f8a7c35f69" (UID: "8685b5d7-4581-4581-9c4f-14f8a7c35f69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:08:23.984175 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:23.984142    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:24.069599 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.069556    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:08:24.069599 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.069592    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7td8\" (UniqueName: \"kubernetes.io/projected/8685b5d7-4581-4581-9c4f-14f8a7c35f69-kube-api-access-q7td8\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:08:24.069599 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.069603    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8685b5d7-4581-4581-9c4f-14f8a7c35f69-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:08:24.069838 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.069611    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8685b5d7-4581-4581-9c4f-14f8a7c35f69-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:08:24.379934 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.379902    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:24.380122 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.379945    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:24.392921 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.392891    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:24.430728 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.430694    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:24.732749 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.732666    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"
Sep 10 20:08:24.732749 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.732669    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9" event={"ID":"8685b5d7-4581-4581-9c4f-14f8a7c35f69","Type":"ContainerDied","Data":"622b455b621a570172c357fb61a6139a97e18cbda4fdc77ab2b4440f62ee36bd"}
Sep 10 20:08:24.732749 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.732739    2561 scope.go:117] "RemoveContainer" containerID="9e43cc4d83fc75e5c3eb94cc74c00f35311f25097b99ffa64bd8c04946ffb74d"
Sep 10 20:08:24.744785 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.744761    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:24.748166 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.748145    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:08:24.757181 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.757149    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"]
Sep 10 20:08:24.763732 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:24.763704    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f563295c-5774ddd6fd-q5fv9"]
Sep 10 20:08:25.019751 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:25.019724    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:25.404253 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:25.404170    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8685b5d7-4581-4581-9c4f-14f8a7c35f69" path="/var/lib/kubelet/pods/8685b5d7-4581-4581-9c4f-14f8a7c35f69/volumes"
Sep 10 20:08:27.441413 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:27.441366    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:27.540079 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:27.540044    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:28.705947 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.705918    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"]
Sep 10 20:08:28.706325 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.706280    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8685b5d7-4581-4581-9c4f-14f8a7c35f69" containerName="maas-api"
Sep 10 20:08:28.706325 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.706291    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8685b5d7-4581-4581-9c4f-14f8a7c35f69" containerName="maas-api"
Sep 10 20:08:28.706447 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.706362    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8685b5d7-4581-4581-9c4f-14f8a7c35f69" containerName="maas-api"
Sep 10 20:08:28.710753 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.710732    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:28.713507 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.713479    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-468cfd8a-serving-cert\""
Sep 10 20:08:28.722259 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.722170    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"]
Sep 10 20:08:28.755453 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.755408    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:28.819807 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.819773    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-metrics-certs\") pod \"maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:28.819978 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.819849    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-maas-api-tls\") pod \"maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:28.819978 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.819882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4560b935-6c95-403d-971c-7fb5838c8fdb-openshift-service-ca\") pod \"maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:28.820052 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.820021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dphml\" (UniqueName: \"kubernetes.io/projected/4560b935-6c95-403d-971c-7fb5838c8fdb-kube-api-access-dphml\") pod \"maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:28.857787 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.857748    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:28.920909 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.920870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-metrics-certs\") pod \"maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:28.921090 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.920920    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-maas-api-tls\") pod \"maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:28.921090 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:08:28.921011    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-serving-cert: secret "maas-api-e2e-ait-468cfd8a-serving-cert" not found
Sep 10 20:08:28.921090 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.921044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4560b935-6c95-403d-971c-7fb5838c8fdb-openshift-service-ca\") pod \"maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:28.921090 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:08:28.921069    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-maas-api-tls podName:4560b935-6c95-403d-971c-7fb5838c8fdb nodeName:}" failed. No retries permitted until 2026-09-10 20:08:29.421053059 +0000 UTC m=+850.471581026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-maas-api-tls") pod "maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r" (UID: "4560b935-6c95-403d-971c-7fb5838c8fdb") : secret "maas-api-e2e-ait-468cfd8a-serving-cert" not found
Sep 10 20:08:28.921329 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.921195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dphml\" (UniqueName: \"kubernetes.io/projected/4560b935-6c95-403d-971c-7fb5838c8fdb-kube-api-access-dphml\") pod \"maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:28.921740 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.921715    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4560b935-6c95-403d-971c-7fb5838c8fdb-openshift-service-ca\") pod \"maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:28.923570 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.923549    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-metrics-certs\") pod \"maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:28.936490 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:28.936467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dphml\" (UniqueName: \"kubernetes.io/projected/4560b935-6c95-403d-971c-7fb5838c8fdb-kube-api-access-dphml\") pod \"maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:29.428031 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:29.427994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-maas-api-tls\") pod \"maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:29.430646 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:29.430623    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-maas-api-tls\") pod \"maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:29.484279 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:29.484243    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:29.614244 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:29.614211    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:29.623171 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:29.623145    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:29.904843 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:29.904819    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"]
Sep 10 20:08:29.906170 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:08:29.906146    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4560b935_6c95_403d_971c_7fb5838c8fdb.slice/crio-e993eefc4bcfaca6826e0adde0f4e056868b96e9d52161746875993d970eb250 WatchSource:0}: Error finding container e993eefc4bcfaca6826e0adde0f4e056868b96e9d52161746875993d970eb250: Status 404 returned error can't find the container with id e993eefc4bcfaca6826e0adde0f4e056868b96e9d52161746875993d970eb250
Sep 10 20:08:30.768549 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:30.768486    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r" event={"ID":"4560b935-6c95-403d-971c-7fb5838c8fdb","Type":"ContainerStarted","Data":"76fa561f713e7c554197f47a3169e96cb486170f791a4bd45b41c9ecf9387569"}
Sep 10 20:08:30.768739 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:30.768595    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:30.768739 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:30.768620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r" event={"ID":"4560b935-6c95-403d-971c-7fb5838c8fdb","Type":"ContainerStarted","Data":"e993eefc4bcfaca6826e0adde0f4e056868b96e9d52161746875993d970eb250"}
Sep 10 20:08:30.787051 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:30.786987    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r" podStartSLOduration=2.438273217 podStartE2EDuration="2.78697055s" totalImagesPullingTime="348.697333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:08:30.785588201 +0000 UTC m=+851.836116191" watchObservedRunningTime="2026-09-10 20:08:30.78697055 +0000 UTC m=+851.837498539"
Sep 10 20:08:36.779021 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:36.778988    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:44.676730 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.676692    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"]
Sep 10 20:08:44.680110 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.680093    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.686147 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.686122    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6d8d62\"/\"openshift-service-ca.crt\""
Sep 10 20:08:44.687200 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.687177    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6d8d62\"/\"test-model-6d8d62-kserve-self-signed-certs\""
Sep 10 20:08:44.687305 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.687211    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6d8d62\"/\"kube-root-ca.crt\""
Sep 10 20:08:44.687305 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.687272    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6d8d62\"/\"default-dockercfg-cxd5p\""
Sep 10 20:08:44.700187 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.700165    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"]
Sep 10 20:08:44.769603 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.769568    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gcj7\" (UniqueName: \"kubernetes.io/projected/8adcd939-74c3-4525-bd1d-4eaafbe9884c-kube-api-access-2gcj7\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.769603 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.769605    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-tmp-dir\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.769846 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.769635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-dshm\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.769846 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.769777    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-home\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.769846 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.769832    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8adcd939-74c3-4525-bd1d-4eaafbe9884c-tls-certs\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.769991 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.769867    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-model-cache\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.794948 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.794901    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:44.870845 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.870795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-home\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.871048 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.870864    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8adcd939-74c3-4525-bd1d-4eaafbe9884c-tls-certs\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.871048 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.870899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-model-cache\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.871048 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.870934    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gcj7\" (UniqueName: \"kubernetes.io/projected/8adcd939-74c3-4525-bd1d-4eaafbe9884c-kube-api-access-2gcj7\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.871048 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.870970    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-tmp-dir\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.871048 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.871006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-dshm\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.871321 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.871204    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-home\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.871321 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.871262    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-model-cache\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.871321 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.871302    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-tmp-dir\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.873490 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.873456    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-dshm\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.873630 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.873609    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8adcd939-74c3-4525-bd1d-4eaafbe9884c-tls-certs\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.880704 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.880681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gcj7\" (UniqueName: \"kubernetes.io/projected/8adcd939-74c3-4525-bd1d-4eaafbe9884c-kube-api-access-2gcj7\") pod \"test-model-6d8d62-kserve-6fc7d88f65-skg45\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") " pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:44.989650 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:44.989558    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:45.274619 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:45.270213    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"]
Sep 10 20:08:45.276496 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:08:45.276455    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8adcd939_74c3_4525_bd1d_4eaafbe9884c.slice/crio-6184030bb18f83981acf72392caf34e237fbc68ef072652d9fe2857afcd98d5d WatchSource:0}: Error finding container 6184030bb18f83981acf72392caf34e237fbc68ef072652d9fe2857afcd98d5d: Status 404 returned error can't find the container with id 6184030bb18f83981acf72392caf34e237fbc68ef072652d9fe2857afcd98d5d
Sep 10 20:08:45.851139 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:45.851098    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45" event={"ID":"8adcd939-74c3-4525-bd1d-4eaafbe9884c","Type":"ContainerStarted","Data":"16582340d12b88cb568cb9baab448e9dc168d7844b8bcc63db8cbd3c44d6a9c2"}
Sep 10 20:08:45.851139 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:45.851150    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45" event={"ID":"8adcd939-74c3-4525-bd1d-4eaafbe9884c","Type":"ContainerStarted","Data":"6184030bb18f83981acf72392caf34e237fbc68ef072652d9fe2857afcd98d5d"}
Sep 10 20:08:45.871973 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:45.871923    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45" podStartSLOduration=1.871903186 podStartE2EDuration="1.871903186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:08:45.871418046 +0000 UTC m=+866.921946035" watchObservedRunningTime="2026-09-10 20:08:45.871903186 +0000 UTC m=+866.922431175"
Sep 10 20:08:47.052079 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:47.052041    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:47.084151 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:47.084111    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"]
Sep 10 20:08:47.154874 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:47.154838    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:47.793631 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:47.793599    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:48.069784 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:48.069695    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:49.955130 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:49.955089    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"]
Sep 10 20:08:49.983188 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:49.983159    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:50.141755 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:50.141719    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:54.989945 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:54.989905    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:54.989945 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:54.989953    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:55.003029 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:55.003005    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:55.916960 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:55.916928    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:08:58.178144 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:58.178115    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:59.245290 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.245257    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"]
Sep 10 20:08:59.245797 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.245508    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r" podUID="4560b935-6c95-403d-971c-7fb5838c8fdb" containerName="maas-api" containerID="cri-o://76fa561f713e7c554197f47a3169e96cb486170f791a4bd45b41c9ecf9387569" gracePeriod=30
Sep 10 20:08:59.291579 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.291551    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:59.401831 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.401798    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:59.496216 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.496138    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:59.517775 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.516949    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:59.529898 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.529866    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"]
Sep 10 20:08:59.530355 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.530322    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn" podUID="3dd2460e-6774-4345-9da6-4b85ea993154" containerName="maas-api" containerID="cri-o://8378a3b4549777a7fb5406a10066374f4fa3734889970f611b06dcd0ec0440af" gracePeriod=30
Sep 10 20:08:59.616479 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.616430    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4560b935-6c95-403d-971c-7fb5838c8fdb-openshift-service-ca\") pod \"4560b935-6c95-403d-971c-7fb5838c8fdb\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") "
Sep 10 20:08:59.616705 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.616514    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-maas-api-tls\") pod \"4560b935-6c95-403d-971c-7fb5838c8fdb\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") "
Sep 10 20:08:59.616705 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.616577    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dphml\" (UniqueName: \"kubernetes.io/projected/4560b935-6c95-403d-971c-7fb5838c8fdb-kube-api-access-dphml\") pod \"4560b935-6c95-403d-971c-7fb5838c8fdb\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") "
Sep 10 20:08:59.616705 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.616617    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-metrics-certs\") pod \"4560b935-6c95-403d-971c-7fb5838c8fdb\" (UID: \"4560b935-6c95-403d-971c-7fb5838c8fdb\") "
Sep 10 20:08:59.616888 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.616870    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4560b935-6c95-403d-971c-7fb5838c8fdb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4560b935-6c95-403d-971c-7fb5838c8fdb" (UID: "4560b935-6c95-403d-971c-7fb5838c8fdb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:08:59.617258 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.617235    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4560b935-6c95-403d-971c-7fb5838c8fdb-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:08:59.619220 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.619193    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4560b935-6c95-403d-971c-7fb5838c8fdb" (UID: "4560b935-6c95-403d-971c-7fb5838c8fdb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:08:59.619834 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.619807    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4560b935-6c95-403d-971c-7fb5838c8fdb" (UID: "4560b935-6c95-403d-971c-7fb5838c8fdb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:08:59.621286 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.621254    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4560b935-6c95-403d-971c-7fb5838c8fdb-kube-api-access-dphml" (OuterVolumeSpecName: "kube-api-access-dphml") pod "4560b935-6c95-403d-971c-7fb5838c8fdb" (UID: "4560b935-6c95-403d-971c-7fb5838c8fdb"). InnerVolumeSpecName "kube-api-access-dphml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:08:59.637600 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.636761    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:59.718186 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.718145    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:08:59.718186 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.718173    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dphml\" (UniqueName: \"kubernetes.io/projected/4560b935-6c95-403d-971c-7fb5838c8fdb-kube-api-access-dphml\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:08:59.718186 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.718183    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4560b935-6c95-403d-971c-7fb5838c8fdb-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:08:59.767939 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.767909    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:59.775769 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.775746    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:08:59.888842 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.888810    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:08:59.919639 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.919602    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dd2460e-6774-4345-9da6-4b85ea993154-metrics-certs\") pod \"3dd2460e-6774-4345-9da6-4b85ea993154\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") "
Sep 10 20:08:59.919797 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.919649    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dd2460e-6774-4345-9da6-4b85ea993154-openshift-service-ca\") pod \"3dd2460e-6774-4345-9da6-4b85ea993154\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") "
Sep 10 20:08:59.919797 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.919702    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dd2460e-6774-4345-9da6-4b85ea993154-maas-api-tls\") pod \"3dd2460e-6774-4345-9da6-4b85ea993154\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") "
Sep 10 20:08:59.919797 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.919722    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chnnb\" (UniqueName: \"kubernetes.io/projected/3dd2460e-6774-4345-9da6-4b85ea993154-kube-api-access-chnnb\") pod \"3dd2460e-6774-4345-9da6-4b85ea993154\" (UID: \"3dd2460e-6774-4345-9da6-4b85ea993154\") "
Sep 10 20:08:59.920083 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.920060    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dd2460e-6774-4345-9da6-4b85ea993154-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3dd2460e-6774-4345-9da6-4b85ea993154" (UID: "3dd2460e-6774-4345-9da6-4b85ea993154"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:08:59.922121 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.922089    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dd2460e-6774-4345-9da6-4b85ea993154-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3dd2460e-6774-4345-9da6-4b85ea993154" (UID: "3dd2460e-6774-4345-9da6-4b85ea993154"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:08:59.922121 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.922112    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dd2460e-6774-4345-9da6-4b85ea993154-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3dd2460e-6774-4345-9da6-4b85ea993154" (UID: "3dd2460e-6774-4345-9da6-4b85ea993154"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:08:59.922259 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.922129    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd2460e-6774-4345-9da6-4b85ea993154-kube-api-access-chnnb" (OuterVolumeSpecName: "kube-api-access-chnnb") pod "3dd2460e-6774-4345-9da6-4b85ea993154" (UID: "3dd2460e-6774-4345-9da6-4b85ea993154"). InnerVolumeSpecName "kube-api-access-chnnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:08:59.929184 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.929159    2561 generic.go:356] "Generic (PLEG): container finished" podID="3dd2460e-6774-4345-9da6-4b85ea993154" containerID="8378a3b4549777a7fb5406a10066374f4fa3734889970f611b06dcd0ec0440af" exitCode=0
Sep 10 20:08:59.929304 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.929207    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"
Sep 10 20:08:59.929304 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.929263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn" event={"ID":"3dd2460e-6774-4345-9da6-4b85ea993154","Type":"ContainerDied","Data":"8378a3b4549777a7fb5406a10066374f4fa3734889970f611b06dcd0ec0440af"}
Sep 10 20:08:59.929406 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.929317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn" event={"ID":"3dd2460e-6774-4345-9da6-4b85ea993154","Type":"ContainerDied","Data":"d5a50b05e10889e73ce6825b2cfac97fad441f9f5b473e098329df397a8682c0"}
Sep 10 20:08:59.929406 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.929343    2561 scope.go:117] "RemoveContainer" containerID="8378a3b4549777a7fb5406a10066374f4fa3734889970f611b06dcd0ec0440af"
Sep 10 20:08:59.930712 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.930692    2561 generic.go:356] "Generic (PLEG): container finished" podID="4560b935-6c95-403d-971c-7fb5838c8fdb" containerID="76fa561f713e7c554197f47a3169e96cb486170f791a4bd45b41c9ecf9387569" exitCode=0
Sep 10 20:08:59.930812 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.930740    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"
Sep 10 20:08:59.930812 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.930775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r" event={"ID":"4560b935-6c95-403d-971c-7fb5838c8fdb","Type":"ContainerDied","Data":"76fa561f713e7c554197f47a3169e96cb486170f791a4bd45b41c9ecf9387569"}
Sep 10 20:08:59.931091 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.930828    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r" event={"ID":"4560b935-6c95-403d-971c-7fb5838c8fdb","Type":"ContainerDied","Data":"e993eefc4bcfaca6826e0adde0f4e056868b96e9d52161746875993d970eb250"}
Sep 10 20:08:59.939789 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.939766    2561 scope.go:117] "RemoveContainer" containerID="8378a3b4549777a7fb5406a10066374f4fa3734889970f611b06dcd0ec0440af"
Sep 10 20:08:59.940203 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:08:59.940165    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8378a3b4549777a7fb5406a10066374f4fa3734889970f611b06dcd0ec0440af\": container with ID starting with 8378a3b4549777a7fb5406a10066374f4fa3734889970f611b06dcd0ec0440af not found: ID does not exist" containerID="8378a3b4549777a7fb5406a10066374f4fa3734889970f611b06dcd0ec0440af"
Sep 10 20:08:59.940364 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.940342    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8378a3b4549777a7fb5406a10066374f4fa3734889970f611b06dcd0ec0440af"} err="failed to get container status \"8378a3b4549777a7fb5406a10066374f4fa3734889970f611b06dcd0ec0440af\": rpc error: code = NotFound desc = could not find container \"8378a3b4549777a7fb5406a10066374f4fa3734889970f611b06dcd0ec0440af\": container with ID starting with 8378a3b4549777a7fb5406a10066374f4fa3734889970f611b06dcd0ec0440af not found: ID does not exist"
Sep 10 20:08:59.940580 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.940560    2561 scope.go:117] "RemoveContainer" containerID="76fa561f713e7c554197f47a3169e96cb486170f791a4bd45b41c9ecf9387569"
Sep 10 20:08:59.950424 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.950404    2561 scope.go:117] "RemoveContainer" containerID="76fa561f713e7c554197f47a3169e96cb486170f791a4bd45b41c9ecf9387569"
Sep 10 20:08:59.950750 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:08:59.950727    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76fa561f713e7c554197f47a3169e96cb486170f791a4bd45b41c9ecf9387569\": container with ID starting with 76fa561f713e7c554197f47a3169e96cb486170f791a4bd45b41c9ecf9387569 not found: ID does not exist" containerID="76fa561f713e7c554197f47a3169e96cb486170f791a4bd45b41c9ecf9387569"
Sep 10 20:08:59.950838 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.950763    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76fa561f713e7c554197f47a3169e96cb486170f791a4bd45b41c9ecf9387569"} err="failed to get container status \"76fa561f713e7c554197f47a3169e96cb486170f791a4bd45b41c9ecf9387569\": rpc error: code = NotFound desc = could not find container \"76fa561f713e7c554197f47a3169e96cb486170f791a4bd45b41c9ecf9387569\": container with ID starting with 76fa561f713e7c554197f47a3169e96cb486170f791a4bd45b41c9ecf9387569 not found: ID does not exist"
Sep 10 20:08:59.957340 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.957319    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"]
Sep 10 20:08:59.961693 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.961672    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-468cfd8a-5446d69c8d-h4b9r"]
Sep 10 20:08:59.973327 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.973304    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"]
Sep 10 20:08:59.982935 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.982886    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-28a5f0-79487fbcfc-ps4mn"]
Sep 10 20:08:59.989663 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:08:59.989638    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:00.021052 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:00.020970    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dd2460e-6774-4345-9da6-4b85ea993154-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:00.021052 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:00.021017    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dd2460e-6774-4345-9da6-4b85ea993154-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:00.021052 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:00.021028    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dd2460e-6774-4345-9da6-4b85ea993154-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:00.021052 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:00.021037    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chnnb\" (UniqueName: \"kubernetes.io/projected/3dd2460e-6774-4345-9da6-4b85ea993154-kube-api-access-chnnb\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:00.325157 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:00.325075    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:00.924507 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:00.924479    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:01.337973 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:01.337936    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:01.404428 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:01.404384    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dd2460e-6774-4345-9da6-4b85ea993154" path="/var/lib/kubelet/pods/3dd2460e-6774-4345-9da6-4b85ea993154/volumes"
Sep 10 20:09:01.404830 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:01.404816    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4560b935-6c95-403d-971c-7fb5838c8fdb" path="/var/lib/kubelet/pods/4560b935-6c95-403d-971c-7fb5838c8fdb/volumes"
Sep 10 20:09:01.686824 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:01.686711    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:01.892839 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:01.892808    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:02.104953 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.104918    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n" podUID="174b6c1f-21c0-4b75-84fc-5765c46f5d24" containerName="main" containerID="cri-o://1fd7de028d5ea32e60cac2e1da5fb3f211e5dfcd6f365f2ecea700ca759faa8d" gracePeriod=45
Sep 10 20:09:02.354803 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.354773    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:09:02.443830 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.443737    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174b6c1f-21c0-4b75-84fc-5765c46f5d24-tls-certs\") pod \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") "
Sep 10 20:09:02.444012 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.443829    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-home\") pod \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") "
Sep 10 20:09:02.444012 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.443865    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-tmp-dir\") pod \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") "
Sep 10 20:09:02.444012 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.443919    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-dshm\") pod \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") "
Sep 10 20:09:02.444012 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.443966    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-model-cache\") pod \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") "
Sep 10 20:09:02.444012 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.443990    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vlnc\" (UniqueName: \"kubernetes.io/projected/174b6c1f-21c0-4b75-84fc-5765c46f5d24-kube-api-access-5vlnc\") pod \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\" (UID: \"174b6c1f-21c0-4b75-84fc-5765c46f5d24\") "
Sep 10 20:09:02.444254 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.444130    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-home" (OuterVolumeSpecName: "home") pod "174b6c1f-21c0-4b75-84fc-5765c46f5d24" (UID: "174b6c1f-21c0-4b75-84fc-5765c46f5d24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:09:02.444254 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.444198    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "174b6c1f-21c0-4b75-84fc-5765c46f5d24" (UID: "174b6c1f-21c0-4b75-84fc-5765c46f5d24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:09:02.444325 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.444246    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-model-cache" (OuterVolumeSpecName: "model-cache") pod "174b6c1f-21c0-4b75-84fc-5765c46f5d24" (UID: "174b6c1f-21c0-4b75-84fc-5765c46f5d24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:09:02.444325 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.444299    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:02.444325 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.444313    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:02.446196 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.446173    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-dshm" (OuterVolumeSpecName: "dshm") pod "174b6c1f-21c0-4b75-84fc-5765c46f5d24" (UID: "174b6c1f-21c0-4b75-84fc-5765c46f5d24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:09:02.446285 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.446220    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174b6c1f-21c0-4b75-84fc-5765c46f5d24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "174b6c1f-21c0-4b75-84fc-5765c46f5d24" (UID: "174b6c1f-21c0-4b75-84fc-5765c46f5d24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:09:02.446323 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.446303    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174b6c1f-21c0-4b75-84fc-5765c46f5d24-kube-api-access-5vlnc" (OuterVolumeSpecName: "kube-api-access-5vlnc") pod "174b6c1f-21c0-4b75-84fc-5765c46f5d24" (UID: "174b6c1f-21c0-4b75-84fc-5765c46f5d24"). InnerVolumeSpecName "kube-api-access-5vlnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:09:02.545358 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.545321    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:02.545358 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.545354    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174b6c1f-21c0-4b75-84fc-5765c46f5d24-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:02.545642 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.545368    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vlnc\" (UniqueName: \"kubernetes.io/projected/174b6c1f-21c0-4b75-84fc-5765c46f5d24-kube-api-access-5vlnc\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:02.545642 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.545384    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174b6c1f-21c0-4b75-84fc-5765c46f5d24-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:02.950560 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.950500    2561 generic.go:356] "Generic (PLEG): container finished" podID="174b6c1f-21c0-4b75-84fc-5765c46f5d24" containerID="1fd7de028d5ea32e60cac2e1da5fb3f211e5dfcd6f365f2ecea700ca759faa8d" exitCode=0
Sep 10 20:09:02.950762 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.950566    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"
Sep 10 20:09:02.950762 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.950575    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n" event={"ID":"174b6c1f-21c0-4b75-84fc-5765c46f5d24","Type":"ContainerDied","Data":"1fd7de028d5ea32e60cac2e1da5fb3f211e5dfcd6f365f2ecea700ca759faa8d"}
Sep 10 20:09:02.950762 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.950615    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n" event={"ID":"174b6c1f-21c0-4b75-84fc-5765c46f5d24","Type":"ContainerDied","Data":"963ad28df15dea6bc28724461291bb52e01aa0f45808bcebd366a8eae06b2ae1"}
Sep 10 20:09:02.950762 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.950635    2561 scope.go:117] "RemoveContainer" containerID="1fd7de028d5ea32e60cac2e1da5fb3f211e5dfcd6f365f2ecea700ca759faa8d"
Sep 10 20:09:02.960040 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.960017    2561 scope.go:117] "RemoveContainer" containerID="1fd7de028d5ea32e60cac2e1da5fb3f211e5dfcd6f365f2ecea700ca759faa8d"
Sep 10 20:09:02.960337 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:09:02.960315    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd7de028d5ea32e60cac2e1da5fb3f211e5dfcd6f365f2ecea700ca759faa8d\": container with ID starting with 1fd7de028d5ea32e60cac2e1da5fb3f211e5dfcd6f365f2ecea700ca759faa8d not found: ID does not exist" containerID="1fd7de028d5ea32e60cac2e1da5fb3f211e5dfcd6f365f2ecea700ca759faa8d"
Sep 10 20:09:02.960385 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.960347    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd7de028d5ea32e60cac2e1da5fb3f211e5dfcd6f365f2ecea700ca759faa8d"} err="failed to get container status \"1fd7de028d5ea32e60cac2e1da5fb3f211e5dfcd6f365f2ecea700ca759faa8d\": rpc error: code = NotFound desc = could not find container \"1fd7de028d5ea32e60cac2e1da5fb3f211e5dfcd6f365f2ecea700ca759faa8d\": container with ID starting with 1fd7de028d5ea32e60cac2e1da5fb3f211e5dfcd6f365f2ecea700ca759faa8d not found: ID does not exist"
Sep 10 20:09:02.973973 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.973943    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"]
Sep 10 20:09:02.976371 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:02.976348    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-6e5a8205-kserve-795fcd7957-vz88n"]
Sep 10 20:09:03.404968 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:03.404937    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="174b6c1f-21c0-4b75-84fc-5765c46f5d24" path="/var/lib/kubelet/pods/174b6c1f-21c0-4b75-84fc-5765c46f5d24/volumes"
Sep 10 20:09:04.409122 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:04.409059    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:04.500588 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:04.500554    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:04.975464 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:04.975426    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml" podUID="aedbd723-f28d-4777-af21-996699b08ec1" containerName="main" containerID="cri-o://58bf4708e895c6af3d9d328e931817a94a7a81e98f58f977b98829477826dad5" gracePeriod=45
Sep 10 20:09:05.230768 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.230705    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:09:05.371973 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.371941    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-tmp-dir\") pod \"aedbd723-f28d-4777-af21-996699b08ec1\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") "
Sep 10 20:09:05.371973 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.371979    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aedbd723-f28d-4777-af21-996699b08ec1-tls-certs\") pod \"aedbd723-f28d-4777-af21-996699b08ec1\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") "
Sep 10 20:09:05.372216 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.372067    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-home\") pod \"aedbd723-f28d-4777-af21-996699b08ec1\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") "
Sep 10 20:09:05.372216 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.372112    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-model-cache\") pod \"aedbd723-f28d-4777-af21-996699b08ec1\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") "
Sep 10 20:09:05.372216 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.372166    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpw5r\" (UniqueName: \"kubernetes.io/projected/aedbd723-f28d-4777-af21-996699b08ec1-kube-api-access-vpw5r\") pod \"aedbd723-f28d-4777-af21-996699b08ec1\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") "
Sep 10 20:09:05.372216 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.372191    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-dshm\") pod \"aedbd723-f28d-4777-af21-996699b08ec1\" (UID: \"aedbd723-f28d-4777-af21-996699b08ec1\") "
Sep 10 20:09:05.372424 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.372249    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aedbd723-f28d-4777-af21-996699b08ec1" (UID: "aedbd723-f28d-4777-af21-996699b08ec1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:09:05.372424 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.372319    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-home" (OuterVolumeSpecName: "home") pod "aedbd723-f28d-4777-af21-996699b08ec1" (UID: "aedbd723-f28d-4777-af21-996699b08ec1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:09:05.372542 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.372425    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-model-cache" (OuterVolumeSpecName: "model-cache") pod "aedbd723-f28d-4777-af21-996699b08ec1" (UID: "aedbd723-f28d-4777-af21-996699b08ec1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:09:05.372596 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.372554    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:05.372596 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.372576    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:05.372596 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.372590    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:05.374371 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.374339    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-dshm" (OuterVolumeSpecName: "dshm") pod "aedbd723-f28d-4777-af21-996699b08ec1" (UID: "aedbd723-f28d-4777-af21-996699b08ec1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:09:05.374371 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.374345    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aedbd723-f28d-4777-af21-996699b08ec1-kube-api-access-vpw5r" (OuterVolumeSpecName: "kube-api-access-vpw5r") pod "aedbd723-f28d-4777-af21-996699b08ec1" (UID: "aedbd723-f28d-4777-af21-996699b08ec1"). InnerVolumeSpecName "kube-api-access-vpw5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:09:05.374515 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.374393    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aedbd723-f28d-4777-af21-996699b08ec1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aedbd723-f28d-4777-af21-996699b08ec1" (UID: "aedbd723-f28d-4777-af21-996699b08ec1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:09:05.473635 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.473594    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpw5r\" (UniqueName: \"kubernetes.io/projected/aedbd723-f28d-4777-af21-996699b08ec1-kube-api-access-vpw5r\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:05.473635 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.473625    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aedbd723-f28d-4777-af21-996699b08ec1-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:05.473635 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.473636    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aedbd723-f28d-4777-af21-996699b08ec1-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:05.528483 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.528453    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"]
Sep 10 20:09:05.528824 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.528812    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174b6c1f-21c0-4b75-84fc-5765c46f5d24" containerName="main"
Sep 10 20:09:05.528892 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.528825    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="174b6c1f-21c0-4b75-84fc-5765c46f5d24" containerName="main"
Sep 10 20:09:05.528892 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.528835    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4560b935-6c95-403d-971c-7fb5838c8fdb" containerName="maas-api"
Sep 10 20:09:05.528892 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.528841    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4560b935-6c95-403d-971c-7fb5838c8fdb" containerName="maas-api"
Sep 10 20:09:05.528892 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.528858    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dd2460e-6774-4345-9da6-4b85ea993154" containerName="maas-api"
Sep 10 20:09:05.528892 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.528863    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dd2460e-6774-4345-9da6-4b85ea993154" containerName="maas-api"
Sep 10 20:09:05.528892 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.528869    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aedbd723-f28d-4777-af21-996699b08ec1" containerName="main"
Sep 10 20:09:05.528892 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.528874    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aedbd723-f28d-4777-af21-996699b08ec1" containerName="main"
Sep 10 20:09:05.529126 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.528934    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dd2460e-6774-4345-9da6-4b85ea993154" containerName="maas-api"
Sep 10 20:09:05.529126 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.528943    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="174b6c1f-21c0-4b75-84fc-5765c46f5d24" containerName="main"
Sep 10 20:09:05.529126 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.528950    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4560b935-6c95-403d-971c-7fb5838c8fdb" containerName="maas-api"
Sep 10 20:09:05.529126 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.528957    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aedbd723-f28d-4777-af21-996699b08ec1" containerName="main"
Sep 10 20:09:05.532118 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.532099    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:05.534694 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.534668    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-653653a4-serving-cert\""
Sep 10 20:09:05.546149 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.546120    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"]
Sep 10 20:09:05.574802 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.574776    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:05.675871 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.675832    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d509851-9eae-4c49-8585-2d87c1de5bc5-openshift-service-ca\") pod \"maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:05.676056 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.675921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p22lc\" (UniqueName: \"kubernetes.io/projected/5d509851-9eae-4c49-8585-2d87c1de5bc5-kube-api-access-p22lc\") pod \"maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:05.676056 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.675946    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-metrics-certs\") pod \"maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:05.676056 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.675982    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-maas-api-tls\") pod \"maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:05.686780 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.681317    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:05.776726 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.776683    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p22lc\" (UniqueName: \"kubernetes.io/projected/5d509851-9eae-4c49-8585-2d87c1de5bc5-kube-api-access-p22lc\") pod \"maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:05.776726 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.776731    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-metrics-certs\") pod \"maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:05.776944 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.776763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-maas-api-tls\") pod \"maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:05.776944 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.776819    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d509851-9eae-4c49-8585-2d87c1de5bc5-openshift-service-ca\") pod \"maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:05.777010 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:09:05.776947    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-serving-cert: secret "maas-api-e2e-derive-653653a4-serving-cert" not found
Sep 10 20:09:05.777045 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:09:05.777015    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-maas-api-tls podName:5d509851-9eae-4c49-8585-2d87c1de5bc5 nodeName:}" failed. No retries permitted until 2026-09-10 20:09:06.27699489 +0000 UTC m=+887.327522860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-maas-api-tls") pod "maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7" (UID: "5d509851-9eae-4c49-8585-2d87c1de5bc5") : secret "maas-api-e2e-derive-653653a4-serving-cert" not found
Sep 10 20:09:05.777522 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.777506    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d509851-9eae-4c49-8585-2d87c1de5bc5-openshift-service-ca\") pod \"maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:05.779383 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.779339    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-metrics-certs\") pod \"maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:05.787396 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.787371    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p22lc\" (UniqueName: \"kubernetes.io/projected/5d509851-9eae-4c49-8585-2d87c1de5bc5-kube-api-access-p22lc\") pod \"maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:05.889151 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.889112    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:05.971953 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.971921    2561 generic.go:356] "Generic (PLEG): container finished" podID="aedbd723-f28d-4777-af21-996699b08ec1" containerID="58bf4708e895c6af3d9d328e931817a94a7a81e98f58f977b98829477826dad5" exitCode=0
Sep 10 20:09:05.972134 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.971976    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"
Sep 10 20:09:05.972134 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.971981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml" event={"ID":"aedbd723-f28d-4777-af21-996699b08ec1","Type":"ContainerDied","Data":"58bf4708e895c6af3d9d328e931817a94a7a81e98f58f977b98829477826dad5"}
Sep 10 20:09:05.972134 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.972012    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml" event={"ID":"aedbd723-f28d-4777-af21-996699b08ec1","Type":"ContainerDied","Data":"1b35c3969470c02ca0bea9dd80447ba918cb901c21a17ff94e3ad29c37c4ae63"}
Sep 10 20:09:05.972134 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.972028    2561 scope.go:117] "RemoveContainer" containerID="58bf4708e895c6af3d9d328e931817a94a7a81e98f58f977b98829477826dad5"
Sep 10 20:09:05.981640 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.981467    2561 scope.go:117] "RemoveContainer" containerID="58bf4708e895c6af3d9d328e931817a94a7a81e98f58f977b98829477826dad5"
Sep 10 20:09:05.981780 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:09:05.981758    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58bf4708e895c6af3d9d328e931817a94a7a81e98f58f977b98829477826dad5\": container with ID starting with 58bf4708e895c6af3d9d328e931817a94a7a81e98f58f977b98829477826dad5 not found: ID does not exist" containerID="58bf4708e895c6af3d9d328e931817a94a7a81e98f58f977b98829477826dad5"
Sep 10 20:09:05.981818 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.981788    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58bf4708e895c6af3d9d328e931817a94a7a81e98f58f977b98829477826dad5"} err="failed to get container status \"58bf4708e895c6af3d9d328e931817a94a7a81e98f58f977b98829477826dad5\": rpc error: code = NotFound desc = could not find container \"58bf4708e895c6af3d9d328e931817a94a7a81e98f58f977b98829477826dad5\": container with ID starting with 58bf4708e895c6af3d9d328e931817a94a7a81e98f58f977b98829477826dad5 not found: ID does not exist"
Sep 10 20:09:05.991129 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.991104    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"]
Sep 10 20:09:05.996658 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:05.996636    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-6e5a8205-kserve-659d74758c-m5tml"]
Sep 10 20:09:06.112607 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:06.112495    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:06.201644 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:06.201609    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:06.281646 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:06.281610    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-maas-api-tls\") pod \"maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:06.284092 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:06.284063    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-maas-api-tls\") pod \"maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:06.442505 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:06.442415    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:06.720723 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:06.720695    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"]
Sep 10 20:09:06.722518 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:09:06.722491    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d509851_9eae_4c49_8585_2d87c1de5bc5.slice/crio-365bffb59f6ef1fbc04bf60bfc0f7626680dac23158606a173c2c8b622ceb108 WatchSource:0}: Error finding container 365bffb59f6ef1fbc04bf60bfc0f7626680dac23158606a173c2c8b622ceb108: Status 404 returned error can't find the container with id 365bffb59f6ef1fbc04bf60bfc0f7626680dac23158606a173c2c8b622ceb108
Sep 10 20:09:06.979880 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:06.979790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7" event={"ID":"5d509851-9eae-4c49-8585-2d87c1de5bc5","Type":"ContainerStarted","Data":"365bffb59f6ef1fbc04bf60bfc0f7626680dac23158606a173c2c8b622ceb108"}
Sep 10 20:09:07.405018 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:07.404986    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aedbd723-f28d-4777-af21-996699b08ec1" path="/var/lib/kubelet/pods/aedbd723-f28d-4777-af21-996699b08ec1/volumes"
Sep 10 20:09:07.986318 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:07.986281    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7" event={"ID":"5d509851-9eae-4c49-8585-2d87c1de5bc5","Type":"ContainerStarted","Data":"86b890d13443e76afca07c204d24b722ab846f1267d3cc5b316ef018303e2952"}
Sep 10 20:09:07.986733 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:07.986414    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:08.008808 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:08.008758    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7" podStartSLOduration=2.539058844 podStartE2EDuration="3.008741495s" totalImagesPullingTime="469.682651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:09:08.007108617 +0000 UTC m=+889.057636642" watchObservedRunningTime="2026-09-10 20:09:08.008741495 +0000 UTC m=+889.059269483"
Sep 10 20:09:08.052167 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:08.052129    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:08.154807 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:08.154776    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:08.344859 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:08.344828    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:08.774461 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:08.774428    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:09.126162 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:09.126086    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:09.306273 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:09.306232    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:12.023428 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:12.023344    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:12.132141 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:12.132109    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:13.147749 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.147716    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"]
Sep 10 20:09:13.151468 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.151451    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:13.154283 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.154264    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-589acd-serving-cert\""
Sep 10 20:09:13.164887 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.164860    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"]
Sep 10 20:09:13.197412 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.197373    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:13.248696 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.248663    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-maas-api-tls\") pod \"maas-api-e2e-explicit-589acd-7476ffb778-f9c4z\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:13.248906 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.248716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8aef37-376e-4b90-b249-fb62121e9953-openshift-service-ca\") pod \"maas-api-e2e-explicit-589acd-7476ffb778-f9c4z\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:13.248906 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.248805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5gjf\" (UniqueName: \"kubernetes.io/projected/6f8aef37-376e-4b90-b249-fb62121e9953-kube-api-access-t5gjf\") pod \"maas-api-e2e-explicit-589acd-7476ffb778-f9c4z\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:13.249003 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.248929    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-metrics-certs\") pod \"maas-api-e2e-explicit-589acd-7476ffb778-f9c4z\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:13.297322 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.297287    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:13.349749 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.349713    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-metrics-certs\") pod \"maas-api-e2e-explicit-589acd-7476ffb778-f9c4z\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:13.349945 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.349787    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-maas-api-tls\") pod \"maas-api-e2e-explicit-589acd-7476ffb778-f9c4z\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:13.349945 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.349824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8aef37-376e-4b90-b249-fb62121e9953-openshift-service-ca\") pod \"maas-api-e2e-explicit-589acd-7476ffb778-f9c4z\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:13.349945 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.349850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5gjf\" (UniqueName: \"kubernetes.io/projected/6f8aef37-376e-4b90-b249-fb62121e9953-kube-api-access-t5gjf\") pod \"maas-api-e2e-explicit-589acd-7476ffb778-f9c4z\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:13.349945 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:09:13.349906    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-serving-cert: secret "maas-api-e2e-explicit-589acd-serving-cert" not found
Sep 10 20:09:13.350158 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:09:13.349998    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-maas-api-tls podName:6f8aef37-376e-4b90-b249-fb62121e9953 nodeName:}" failed. No retries permitted until 2026-09-10 20:09:13.849976877 +0000 UTC m=+894.900504872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-maas-api-tls") pod "maas-api-e2e-explicit-589acd-7476ffb778-f9c4z" (UID: "6f8aef37-376e-4b90-b249-fb62121e9953") : secret "maas-api-e2e-explicit-589acd-serving-cert" not found
Sep 10 20:09:13.350493 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.350472    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8aef37-376e-4b90-b249-fb62121e9953-openshift-service-ca\") pod \"maas-api-e2e-explicit-589acd-7476ffb778-f9c4z\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:13.352403 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.352383    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-metrics-certs\") pod \"maas-api-e2e-explicit-589acd-7476ffb778-f9c4z\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:13.367380 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.367351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5gjf\" (UniqueName: \"kubernetes.io/projected/6f8aef37-376e-4b90-b249-fb62121e9953-kube-api-access-t5gjf\") pod \"maas-api-e2e-explicit-589acd-7476ffb778-f9c4z\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:13.773261 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.773230    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:13.856085 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.856046    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-maas-api-tls\") pod \"maas-api-e2e-explicit-589acd-7476ffb778-f9c4z\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:13.858752 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.858731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-maas-api-tls\") pod \"maas-api-e2e-explicit-589acd-7476ffb778-f9c4z\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:13.873671 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.873639    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:13.978121 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.978093    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:13.998369 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:13.998341    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:14.063706 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:14.063617    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:14.343580 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:14.343554    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"]
Sep 10 20:09:14.345360 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:09:14.345336    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8aef37_376e_4b90_b249_fb62121e9953.slice/crio-7aae8315f41b8131635b1a3918c950d180139ac789a0c4a40fdd57702f70b6b5 WatchSource:0}: Error finding container 7aae8315f41b8131635b1a3918c950d180139ac789a0c4a40fdd57702f70b6b5: Status 404 returned error can't find the container with id 7aae8315f41b8131635b1a3918c950d180139ac789a0c4a40fdd57702f70b6b5
Sep 10 20:09:15.027672 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:15.027632    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z" event={"ID":"6f8aef37-376e-4b90-b249-fb62121e9953","Type":"ContainerStarted","Data":"01ed84aa4260b20964f240a63f111e994b8672f9a009a1b9bf02efca7eba8fd7"}
Sep 10 20:09:15.027868 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:15.027698    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z" event={"ID":"6f8aef37-376e-4b90-b249-fb62121e9953","Type":"ContainerStarted","Data":"7aae8315f41b8131635b1a3918c950d180139ac789a0c4a40fdd57702f70b6b5"}
Sep 10 20:09:15.027868 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:15.027740    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:15.049621 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:15.049566    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z" podStartSLOduration=1.5730358039999999 podStartE2EDuration="2.049548822s" totalImagesPullingTime="476.513018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:09:15.045731353 +0000 UTC m=+896.096259342" watchObservedRunningTime="2026-09-10 20:09:15.049548822 +0000 UTC m=+896.100076802"
Sep 10 20:09:20.809069 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:20.808957    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 20:09:20.826271 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:20.811706    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 20:09:21.041560 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:21.040171    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:09:26.640640 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:26.640610    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:30.659113 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.659067    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"]
Sep 10 20:09:30.662482 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.662463    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.666738 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.666710    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-39487c\"/\"test-model-39487c-kserve-self-signed-certs\""
Sep 10 20:09:30.667494 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.667478    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-39487c\"/\"kube-root-ca.crt\""
Sep 10 20:09:30.667586 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.667511    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-39487c\"/\"openshift-service-ca.crt\""
Sep 10 20:09:30.669743 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.669728    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-39487c\"/\"default-dockercfg-9tpwb\""
Sep 10 20:09:30.685380 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.685354    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"]
Sep 10 20:09:30.707881 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.707848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-tmp-dir\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.708035 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.707893    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cca45dd-4408-4d5e-8307-59864450eac8-tls-certs\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.708035 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.707913    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44g9\" (UniqueName: \"kubernetes.io/projected/4cca45dd-4408-4d5e-8307-59864450eac8-kube-api-access-g44g9\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.708035 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.707992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-dshm\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.708035 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.708020    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-home\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.708176 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.708045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-model-cache\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.809628 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.809593    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-tmp-dir\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.809628 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.809634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cca45dd-4408-4d5e-8307-59864450eac8-tls-certs\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.809888 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.809655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44g9\" (UniqueName: \"kubernetes.io/projected/4cca45dd-4408-4d5e-8307-59864450eac8-kube-api-access-g44g9\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.809888 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.809728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-dshm\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.809888 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.809750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-home\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.809888 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.809772    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-model-cache\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.810074 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.810053    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-tmp-dir\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.810152 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.810079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-home\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.810214 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.810193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-model-cache\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.812068 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.812050    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-dshm\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.812246 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.812226    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cca45dd-4408-4d5e-8307-59864450eac8-tls-certs\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.816360 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.816312    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:30.821113 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.821094    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44g9\" (UniqueName: \"kubernetes.io/projected/4cca45dd-4408-4d5e-8307-59864450eac8-kube-api-access-g44g9\") pod \"test-model-39487c-kserve-696789c7b5-bbh46\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") " pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:30.972095 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:30.971994    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:31.254916 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:31.254894    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"]
Sep 10 20:09:31.256062 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:09:31.256038    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cca45dd_4408_4d5e_8307_59864450eac8.slice/crio-b9d5e15030e61e18d3b9863671b3c9621de54d3c92ef9825de0442087c3c6305 WatchSource:0}: Error finding container b9d5e15030e61e18d3b9863671b3c9621de54d3c92ef9825de0442087c3c6305: Status 404 returned error can't find the container with id b9d5e15030e61e18d3b9863671b3c9621de54d3c92ef9825de0442087c3c6305
Sep 10 20:09:32.119095 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:32.119060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46" event={"ID":"4cca45dd-4408-4d5e-8307-59864450eac8","Type":"ContainerStarted","Data":"f14c009396fc0299f7c6e9a05d9071c98e5d4fb9426dec0e57d7f58883a8e853"}
Sep 10 20:09:32.119095 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:32.119102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46" event={"ID":"4cca45dd-4408-4d5e-8307-59864450eac8","Type":"ContainerStarted","Data":"b9d5e15030e61e18d3b9863671b3c9621de54d3c92ef9825de0442087c3c6305"}
Sep 10 20:09:32.141721 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:32.141662    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46" podStartSLOduration=2.141644048 podStartE2EDuration="2.141644048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:09:32.139682466 +0000 UTC m=+913.190210472" watchObservedRunningTime="2026-09-10 20:09:32.141644048 +0000 UTC m=+913.192172037"
Sep 10 20:09:34.314894 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:34.314849    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:35.684322 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:35.684292    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"]
Sep 10 20:09:35.684800 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:35.684508    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7" podUID="5d509851-9eae-4c49-8585-2d87c1de5bc5" containerName="maas-api" containerID="cri-o://86b890d13443e76afca07c204d24b722ab846f1267d3cc5b316ef018303e2952" gracePeriod=30
Sep 10 20:09:35.723977 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:35.723946    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:35.841226 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:35.841191    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:35.935870 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:35.935801    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:35.952842 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:35.952808    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:36.059108 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.059075    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d509851-9eae-4c49-8585-2d87c1de5bc5-openshift-service-ca\") pod \"5d509851-9eae-4c49-8585-2d87c1de5bc5\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") "
Sep 10 20:09:36.059108 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.059121    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-maas-api-tls\") pod \"5d509851-9eae-4c49-8585-2d87c1de5bc5\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") "
Sep 10 20:09:36.059375 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.059170    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-metrics-certs\") pod \"5d509851-9eae-4c49-8585-2d87c1de5bc5\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") "
Sep 10 20:09:36.059375 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.059222    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p22lc\" (UniqueName: \"kubernetes.io/projected/5d509851-9eae-4c49-8585-2d87c1de5bc5-kube-api-access-p22lc\") pod \"5d509851-9eae-4c49-8585-2d87c1de5bc5\" (UID: \"5d509851-9eae-4c49-8585-2d87c1de5bc5\") "
Sep 10 20:09:36.059574 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.059547    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d509851-9eae-4c49-8585-2d87c1de5bc5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d509851-9eae-4c49-8585-2d87c1de5bc5" (UID: "5d509851-9eae-4c49-8585-2d87c1de5bc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:09:36.060376 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.060350    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:36.061622 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.061599    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d509851-9eae-4c49-8585-2d87c1de5bc5-kube-api-access-p22lc" (OuterVolumeSpecName: "kube-api-access-p22lc") pod "5d509851-9eae-4c49-8585-2d87c1de5bc5" (UID: "5d509851-9eae-4c49-8585-2d87c1de5bc5"). InnerVolumeSpecName "kube-api-access-p22lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:09:36.062022 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.062000    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d509851-9eae-4c49-8585-2d87c1de5bc5" (UID: "5d509851-9eae-4c49-8585-2d87c1de5bc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:09:36.062117 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.062023    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5d509851-9eae-4c49-8585-2d87c1de5bc5" (UID: "5d509851-9eae-4c49-8585-2d87c1de5bc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:09:36.142009 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.141971    2561 generic.go:356] "Generic (PLEG): container finished" podID="5d509851-9eae-4c49-8585-2d87c1de5bc5" containerID="86b890d13443e76afca07c204d24b722ab846f1267d3cc5b316ef018303e2952" exitCode=0
Sep 10 20:09:36.142174 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.142019    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"
Sep 10 20:09:36.142174 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.142059    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7" event={"ID":"5d509851-9eae-4c49-8585-2d87c1de5bc5","Type":"ContainerDied","Data":"86b890d13443e76afca07c204d24b722ab846f1267d3cc5b316ef018303e2952"}
Sep 10 20:09:36.142174 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.142118    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7" event={"ID":"5d509851-9eae-4c49-8585-2d87c1de5bc5","Type":"ContainerDied","Data":"365bffb59f6ef1fbc04bf60bfc0f7626680dac23158606a173c2c8b622ceb108"}
Sep 10 20:09:36.142174 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.142142    2561 scope.go:117] "RemoveContainer" containerID="86b890d13443e76afca07c204d24b722ab846f1267d3cc5b316ef018303e2952"
Sep 10 20:09:36.152588 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.152567    2561 scope.go:117] "RemoveContainer" containerID="86b890d13443e76afca07c204d24b722ab846f1267d3cc5b316ef018303e2952"
Sep 10 20:09:36.152863 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:09:36.152847    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b890d13443e76afca07c204d24b722ab846f1267d3cc5b316ef018303e2952\": container with ID starting with 86b890d13443e76afca07c204d24b722ab846f1267d3cc5b316ef018303e2952 not found: ID does not exist" containerID="86b890d13443e76afca07c204d24b722ab846f1267d3cc5b316ef018303e2952"
Sep 10 20:09:36.152911 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.152872    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b890d13443e76afca07c204d24b722ab846f1267d3cc5b316ef018303e2952"} err="failed to get container status \"86b890d13443e76afca07c204d24b722ab846f1267d3cc5b316ef018303e2952\": rpc error: code = NotFound desc = could not find container \"86b890d13443e76afca07c204d24b722ab846f1267d3cc5b316ef018303e2952\": container with ID starting with 86b890d13443e76afca07c204d24b722ab846f1267d3cc5b316ef018303e2952 not found: ID does not exist"
Sep 10 20:09:36.160867 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.160845    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:36.160867 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.160867    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p22lc\" (UniqueName: \"kubernetes.io/projected/5d509851-9eae-4c49-8585-2d87c1de5bc5-kube-api-access-p22lc\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:36.161010 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.160877    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d509851-9eae-4c49-8585-2d87c1de5bc5-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:36.161010 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.160885    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d509851-9eae-4c49-8585-2d87c1de5bc5-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:09:36.170789 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.170765    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"]
Sep 10 20:09:36.170919 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.170893    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:36.175306 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.175279    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-653653a4-5c5cb779bb-mzkx7"]
Sep 10 20:09:36.428264 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:36.428230    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:37.142509 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:37.142478    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:37.252150 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:37.252110    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:37.362566 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:37.362500    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:37.403960 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:37.403887    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d509851-9eae-4c49-8585-2d87c1de5bc5" path="/var/lib/kubelet/pods/5d509851-9eae-4c49-8585-2d87c1de5bc5/volumes"
Sep 10 20:09:38.859184 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:38.859136    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:40.972219 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:40.972177    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:40.972694 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:40.972243    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:40.997092 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:40.997055    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:41.180693 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:41.180663    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:09:44.923548 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:44.923499    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:45.031244 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:45.031211    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:45.135508 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:45.135472    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:45.356774 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:45.356741    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:45.685268 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:45.685180    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:09:58.758546 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:09:58.758499    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:11.716261 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:11.716224    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:18.332122 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:18.332086    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:19.636094 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:19.636034    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:19.822505 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:19.822467    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:19.956852 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:19.956816    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:29.236138 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.236104    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"]
Sep 10 20:10:29.236596 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.236304    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z" podUID="6f8aef37-376e-4b90-b249-fb62121e9953" containerName="maas-api" containerID="cri-o://01ed84aa4260b20964f240a63f111e994b8672f9a009a1b9bf02efca7eba8fd7" gracePeriod=30
Sep 10 20:10:29.295728 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.295670    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:29.427062 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.427033    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f8aef37-376e-4b90-b249-fb62121e9953" containerID="01ed84aa4260b20964f240a63f111e994b8672f9a009a1b9bf02efca7eba8fd7" exitCode=0
Sep 10 20:10:29.427230 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.427109    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z" event={"ID":"6f8aef37-376e-4b90-b249-fb62121e9953","Type":"ContainerDied","Data":"01ed84aa4260b20964f240a63f111e994b8672f9a009a1b9bf02efca7eba8fd7"}
Sep 10 20:10:29.428611 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.428580    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:29.490223 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.490166    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:10:29.551121 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.551091    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8aef37-376e-4b90-b249-fb62121e9953-openshift-service-ca\") pod \"6f8aef37-376e-4b90-b249-fb62121e9953\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") "
Sep 10 20:10:29.551315 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.551169    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5gjf\" (UniqueName: \"kubernetes.io/projected/6f8aef37-376e-4b90-b249-fb62121e9953-kube-api-access-t5gjf\") pod \"6f8aef37-376e-4b90-b249-fb62121e9953\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") "
Sep 10 20:10:29.551315 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.551193    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-metrics-certs\") pod \"6f8aef37-376e-4b90-b249-fb62121e9953\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") "
Sep 10 20:10:29.551471 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.551313    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-maas-api-tls\") pod \"6f8aef37-376e-4b90-b249-fb62121e9953\" (UID: \"6f8aef37-376e-4b90-b249-fb62121e9953\") "
Sep 10 20:10:29.551523 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.551486    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8aef37-376e-4b90-b249-fb62121e9953-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f8aef37-376e-4b90-b249-fb62121e9953" (UID: "6f8aef37-376e-4b90-b249-fb62121e9953"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:10:29.551655 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.551631    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8aef37-376e-4b90-b249-fb62121e9953-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:29.553632 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.553605    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6f8aef37-376e-4b90-b249-fb62121e9953" (UID: "6f8aef37-376e-4b90-b249-fb62121e9953"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:10:29.553769 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.553743    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8aef37-376e-4b90-b249-fb62121e9953-kube-api-access-t5gjf" (OuterVolumeSpecName: "kube-api-access-t5gjf") pod "6f8aef37-376e-4b90-b249-fb62121e9953" (UID: "6f8aef37-376e-4b90-b249-fb62121e9953"). InnerVolumeSpecName "kube-api-access-t5gjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:10:29.553905 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.553878    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f8aef37-376e-4b90-b249-fb62121e9953" (UID: "6f8aef37-376e-4b90-b249-fb62121e9953"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:10:29.581160 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.581130    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:29.652232 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.652198    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5gjf\" (UniqueName: \"kubernetes.io/projected/6f8aef37-376e-4b90-b249-fb62121e9953-kube-api-access-t5gjf\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:29.652232 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.652229    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:29.652232 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.652239    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f8aef37-376e-4b90-b249-fb62121e9953-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:29.820933 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.820886    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:29.977024 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:29.976992    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:30.149715 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:30.149622    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:30.434034 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:30.433947    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"
Sep 10 20:10:30.434034 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:30.433970    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z" event={"ID":"6f8aef37-376e-4b90-b249-fb62121e9953","Type":"ContainerDied","Data":"7aae8315f41b8131635b1a3918c950d180139ac789a0c4a40fdd57702f70b6b5"}
Sep 10 20:10:30.434034 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:30.434030    2561 scope.go:117] "RemoveContainer" containerID="01ed84aa4260b20964f240a63f111e994b8672f9a009a1b9bf02efca7eba8fd7"
Sep 10 20:10:30.465149 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:30.465109    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"]
Sep 10 20:10:30.470770 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:30.470737    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-589acd-7476ffb778-f9c4z"]
Sep 10 20:10:31.404080 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:31.404047    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f8aef37-376e-4b90-b249-fb62121e9953" path="/var/lib/kubelet/pods/6f8aef37-376e-4b90-b249-fb62121e9953/volumes"
Sep 10 20:10:35.329438 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:35.329402    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"]
Sep 10 20:10:35.400605 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:35.400573    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:35.851662 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:35.851627    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:36.594770 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:36.594726    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"]
Sep 10 20:10:36.638251 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:36.638212    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:36.926025 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:36.925948    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:40.033265 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:40.033224    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:40.136477 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:40.136437    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:40.359828 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:40.359730    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:40.816088 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:40.816043    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:41.288105 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:41.288072    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:41.581636 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:41.581550    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:42.293943 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:42.293909    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:42.448658 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:42.448620    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:42.650714 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:42.650626    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:44.648134 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:44.648100    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:44.803802 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:44.803770    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:45.056573 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:45.056511    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:47.899714 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:47.899680    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"]
Sep 10 20:10:47.900240 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:47.899946    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl" podUID="051f910f-86bf-49fa-93ed-9189bdc21f17" containerName="maas-api" containerID="cri-o://b4cd53c17463f61bae7aa36972eb840b40e9513e58d0214b7c89721193579ae3" gracePeriod=30
Sep 10 20:10:47.992050 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:47.992022    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:48.208236 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:48.208147    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:48.443349 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:48.443304    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:48.615335 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:48.615307    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:49.013998 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.013965    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:49.147884 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.147857    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:10:49.233332 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.233241    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f910f-86bf-49fa-93ed-9189bdc21f17-openshift-service-ca\") pod \"051f910f-86bf-49fa-93ed-9189bdc21f17\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") "
Sep 10 20:10:49.233332 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.233289    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-maas-api-tls\") pod \"051f910f-86bf-49fa-93ed-9189bdc21f17\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") "
Sep 10 20:10:49.233332 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.233313    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-metrics-certs\") pod \"051f910f-86bf-49fa-93ed-9189bdc21f17\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") "
Sep 10 20:10:49.233775 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.233382    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cl7k\" (UniqueName: \"kubernetes.io/projected/051f910f-86bf-49fa-93ed-9189bdc21f17-kube-api-access-2cl7k\") pod \"051f910f-86bf-49fa-93ed-9189bdc21f17\" (UID: \"051f910f-86bf-49fa-93ed-9189bdc21f17\") "
Sep 10 20:10:49.233775 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.233713    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051f910f-86bf-49fa-93ed-9189bdc21f17-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "051f910f-86bf-49fa-93ed-9189bdc21f17" (UID: "051f910f-86bf-49fa-93ed-9189bdc21f17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:10:49.235735 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.235711    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "051f910f-86bf-49fa-93ed-9189bdc21f17" (UID: "051f910f-86bf-49fa-93ed-9189bdc21f17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:10:49.235847 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.235748    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "051f910f-86bf-49fa-93ed-9189bdc21f17" (UID: "051f910f-86bf-49fa-93ed-9189bdc21f17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:10:49.235847 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.235819    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051f910f-86bf-49fa-93ed-9189bdc21f17-kube-api-access-2cl7k" (OuterVolumeSpecName: "kube-api-access-2cl7k") pod "051f910f-86bf-49fa-93ed-9189bdc21f17" (UID: "051f910f-86bf-49fa-93ed-9189bdc21f17"). InnerVolumeSpecName "kube-api-access-2cl7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:10:49.334584 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.334517    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f910f-86bf-49fa-93ed-9189bdc21f17-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:49.334584 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.334582    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:49.334584 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.334592    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f910f-86bf-49fa-93ed-9189bdc21f17-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:49.334919 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.334600    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cl7k\" (UniqueName: \"kubernetes.io/projected/051f910f-86bf-49fa-93ed-9189bdc21f17-kube-api-access-2cl7k\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:49.541255 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.541221    2561 generic.go:356] "Generic (PLEG): container finished" podID="051f910f-86bf-49fa-93ed-9189bdc21f17" containerID="b4cd53c17463f61bae7aa36972eb840b40e9513e58d0214b7c89721193579ae3" exitCode=0
Sep 10 20:10:49.541429 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.541262    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"
Sep 10 20:10:49.541429 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.541307    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl" event={"ID":"051f910f-86bf-49fa-93ed-9189bdc21f17","Type":"ContainerDied","Data":"b4cd53c17463f61bae7aa36972eb840b40e9513e58d0214b7c89721193579ae3"}
Sep 10 20:10:49.541429 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.541355    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl" event={"ID":"051f910f-86bf-49fa-93ed-9189bdc21f17","Type":"ContainerDied","Data":"7644764f8e1a397cd05f802ff341717155d7991583fc097137e876ec211f230a"}
Sep 10 20:10:49.541429 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.541382    2561 scope.go:117] "RemoveContainer" containerID="b4cd53c17463f61bae7aa36972eb840b40e9513e58d0214b7c89721193579ae3"
Sep 10 20:10:49.550364 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.550346    2561 scope.go:117] "RemoveContainer" containerID="b4cd53c17463f61bae7aa36972eb840b40e9513e58d0214b7c89721193579ae3"
Sep 10 20:10:49.550684 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:10:49.550663    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4cd53c17463f61bae7aa36972eb840b40e9513e58d0214b7c89721193579ae3\": container with ID starting with b4cd53c17463f61bae7aa36972eb840b40e9513e58d0214b7c89721193579ae3 not found: ID does not exist" containerID="b4cd53c17463f61bae7aa36972eb840b40e9513e58d0214b7c89721193579ae3"
Sep 10 20:10:49.550748 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.550693    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4cd53c17463f61bae7aa36972eb840b40e9513e58d0214b7c89721193579ae3"} err="failed to get container status \"b4cd53c17463f61bae7aa36972eb840b40e9513e58d0214b7c89721193579ae3\": rpc error: code = NotFound desc = could not find container \"b4cd53c17463f61bae7aa36972eb840b40e9513e58d0214b7c89721193579ae3\": container with ID starting with b4cd53c17463f61bae7aa36972eb840b40e9513e58d0214b7c89721193579ae3 not found: ID does not exist"
Sep 10 20:10:49.586907 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.586877    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"]
Sep 10 20:10:49.592846 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:49.592822    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6d8d62-65d94c695f-n6mnl"]
Sep 10 20:10:50.349697 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.349652    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45" podUID="8adcd939-74c3-4525-bd1d-4eaafbe9884c" containerName="main" containerID="cri-o://16582340d12b88cb568cb9baab448e9dc168d7844b8bcc63db8cbd3c44d6a9c2" gracePeriod=45
Sep 10 20:10:50.548577 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.548549    2561 generic.go:356] "Generic (PLEG): container finished" podID="8adcd939-74c3-4525-bd1d-4eaafbe9884c" containerID="16582340d12b88cb568cb9baab448e9dc168d7844b8bcc63db8cbd3c44d6a9c2" exitCode=0
Sep 10 20:10:50.548737 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.548633    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45" event={"ID":"8adcd939-74c3-4525-bd1d-4eaafbe9884c","Type":"ContainerDied","Data":"16582340d12b88cb568cb9baab448e9dc168d7844b8bcc63db8cbd3c44d6a9c2"}
Sep 10 20:10:50.601556 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.601476    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:10:50.644716 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.644689    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-home\") pod \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") "
Sep 10 20:10:50.644907 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.644729    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gcj7\" (UniqueName: \"kubernetes.io/projected/8adcd939-74c3-4525-bd1d-4eaafbe9884c-kube-api-access-2gcj7\") pod \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") "
Sep 10 20:10:50.644907 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.644760    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-model-cache\") pod \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") "
Sep 10 20:10:50.644907 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.644800    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-dshm\") pod \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") "
Sep 10 20:10:50.644907 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.644837    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8adcd939-74c3-4525-bd1d-4eaafbe9884c-tls-certs\") pod \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") "
Sep 10 20:10:50.645087 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.644938    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-tmp-dir\") pod \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\" (UID: \"8adcd939-74c3-4525-bd1d-4eaafbe9884c\") "
Sep 10 20:10:50.645087 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.645051    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-home" (OuterVolumeSpecName: "home") pod "8adcd939-74c3-4525-bd1d-4eaafbe9884c" (UID: "8adcd939-74c3-4525-bd1d-4eaafbe9884c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:10:50.645250 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.645228    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:50.645403 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.645285    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8adcd939-74c3-4525-bd1d-4eaafbe9884c" (UID: "8adcd939-74c3-4525-bd1d-4eaafbe9884c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:10:50.645473 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.645371    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8adcd939-74c3-4525-bd1d-4eaafbe9884c" (UID: "8adcd939-74c3-4525-bd1d-4eaafbe9884c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:10:50.647516 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.647487    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8adcd939-74c3-4525-bd1d-4eaafbe9884c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8adcd939-74c3-4525-bd1d-4eaafbe9884c" (UID: "8adcd939-74c3-4525-bd1d-4eaafbe9884c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:10:50.647516 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.647508    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-dshm" (OuterVolumeSpecName: "dshm") pod "8adcd939-74c3-4525-bd1d-4eaafbe9884c" (UID: "8adcd939-74c3-4525-bd1d-4eaafbe9884c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:10:50.647732 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.647517    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8adcd939-74c3-4525-bd1d-4eaafbe9884c-kube-api-access-2gcj7" (OuterVolumeSpecName: "kube-api-access-2gcj7") pod "8adcd939-74c3-4525-bd1d-4eaafbe9884c" (UID: "8adcd939-74c3-4525-bd1d-4eaafbe9884c"). InnerVolumeSpecName "kube-api-access-2gcj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:10:50.746600 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.746565    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:50.746600 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.746594    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8adcd939-74c3-4525-bd1d-4eaafbe9884c-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:50.746600 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.746604    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:50.746843 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.746613    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gcj7\" (UniqueName: \"kubernetes.io/projected/8adcd939-74c3-4525-bd1d-4eaafbe9884c-kube-api-access-2gcj7\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:50.746843 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:50.746624    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8adcd939-74c3-4525-bd1d-4eaafbe9884c-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:51.404768 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.404733    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="051f910f-86bf-49fa-93ed-9189bdc21f17" path="/var/lib/kubelet/pods/051f910f-86bf-49fa-93ed-9189bdc21f17/volumes"
Sep 10 20:10:51.556513 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.556481    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"
Sep 10 20:10:51.556513 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.556503    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45" event={"ID":"8adcd939-74c3-4525-bd1d-4eaafbe9884c","Type":"ContainerDied","Data":"6184030bb18f83981acf72392caf34e237fbc68ef072652d9fe2857afcd98d5d"}
Sep 10 20:10:51.556772 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.556567    2561 scope.go:117] "RemoveContainer" containerID="16582340d12b88cb568cb9baab448e9dc168d7844b8bcc63db8cbd3c44d6a9c2"
Sep 10 20:10:51.582071 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.582034    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"]
Sep 10 20:10:51.587283 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.587250    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6d8d62/test-model-6d8d62-kserve-6fc7d88f65-skg45"]
Sep 10 20:10:51.614950 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.614917    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46" podUID="4cca45dd-4408-4d5e-8307-59864450eac8" containerName="main" containerID="cri-o://f14c009396fc0299f7c6e9a05d9071c98e5d4fb9426dec0e57d7f58883a8e853" gracePeriod=45
Sep 10 20:10:51.867438 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.867408    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:10:51.956584 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.956551    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cca45dd-4408-4d5e-8307-59864450eac8-tls-certs\") pod \"4cca45dd-4408-4d5e-8307-59864450eac8\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") "
Sep 10 20:10:51.956584 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.956591    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-dshm\") pod \"4cca45dd-4408-4d5e-8307-59864450eac8\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") "
Sep 10 20:10:51.956817 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.956608    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-model-cache\") pod \"4cca45dd-4408-4d5e-8307-59864450eac8\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") "
Sep 10 20:10:51.956817 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.956627    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-home\") pod \"4cca45dd-4408-4d5e-8307-59864450eac8\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") "
Sep 10 20:10:51.956817 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.956674    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g44g9\" (UniqueName: \"kubernetes.io/projected/4cca45dd-4408-4d5e-8307-59864450eac8-kube-api-access-g44g9\") pod \"4cca45dd-4408-4d5e-8307-59864450eac8\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") "
Sep 10 20:10:51.956817 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.956731    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-tmp-dir\") pod \"4cca45dd-4408-4d5e-8307-59864450eac8\" (UID: \"4cca45dd-4408-4d5e-8307-59864450eac8\") "
Sep 10 20:10:51.957048 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.956890    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-model-cache" (OuterVolumeSpecName: "model-cache") pod "4cca45dd-4408-4d5e-8307-59864450eac8" (UID: "4cca45dd-4408-4d5e-8307-59864450eac8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:10:51.957048 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.956903    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-home" (OuterVolumeSpecName: "home") pod "4cca45dd-4408-4d5e-8307-59864450eac8" (UID: "4cca45dd-4408-4d5e-8307-59864450eac8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:10:51.957141 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.957079    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4cca45dd-4408-4d5e-8307-59864450eac8" (UID: "4cca45dd-4408-4d5e-8307-59864450eac8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:10:51.957141 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.957086    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:51.957141 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.957110    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:51.959156 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.959130    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-dshm" (OuterVolumeSpecName: "dshm") pod "4cca45dd-4408-4d5e-8307-59864450eac8" (UID: "4cca45dd-4408-4d5e-8307-59864450eac8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:10:51.959156 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.959144    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cca45dd-4408-4d5e-8307-59864450eac8-kube-api-access-g44g9" (OuterVolumeSpecName: "kube-api-access-g44g9") pod "4cca45dd-4408-4d5e-8307-59864450eac8" (UID: "4cca45dd-4408-4d5e-8307-59864450eac8"). InnerVolumeSpecName "kube-api-access-g44g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:10:51.961634 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:51.959138    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cca45dd-4408-4d5e-8307-59864450eac8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4cca45dd-4408-4d5e-8307-59864450eac8" (UID: "4cca45dd-4408-4d5e-8307-59864450eac8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:10:52.057948 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:52.057902    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cca45dd-4408-4d5e-8307-59864450eac8-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:52.057948 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:52.057937    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:52.057948 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:52.057951    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g44g9\" (UniqueName: \"kubernetes.io/projected/4cca45dd-4408-4d5e-8307-59864450eac8-kube-api-access-g44g9\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:52.058201 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:52.057987    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cca45dd-4408-4d5e-8307-59864450eac8-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:10:52.563805 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:52.563771    2561 generic.go:356] "Generic (PLEG): container finished" podID="4cca45dd-4408-4d5e-8307-59864450eac8" containerID="f14c009396fc0299f7c6e9a05d9071c98e5d4fb9426dec0e57d7f58883a8e853" exitCode=0
Sep 10 20:10:52.564237 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:52.563829    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"
Sep 10 20:10:52.564237 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:52.563841    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46" event={"ID":"4cca45dd-4408-4d5e-8307-59864450eac8","Type":"ContainerDied","Data":"f14c009396fc0299f7c6e9a05d9071c98e5d4fb9426dec0e57d7f58883a8e853"}
Sep 10 20:10:52.564237 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:52.563872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46" event={"ID":"4cca45dd-4408-4d5e-8307-59864450eac8","Type":"ContainerDied","Data":"b9d5e15030e61e18d3b9863671b3c9621de54d3c92ef9825de0442087c3c6305"}
Sep 10 20:10:52.564237 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:52.563887    2561 scope.go:117] "RemoveContainer" containerID="f14c009396fc0299f7c6e9a05d9071c98e5d4fb9426dec0e57d7f58883a8e853"
Sep 10 20:10:52.573090 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:52.573070    2561 scope.go:117] "RemoveContainer" containerID="f14c009396fc0299f7c6e9a05d9071c98e5d4fb9426dec0e57d7f58883a8e853"
Sep 10 20:10:52.573365 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:10:52.573346    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14c009396fc0299f7c6e9a05d9071c98e5d4fb9426dec0e57d7f58883a8e853\": container with ID starting with f14c009396fc0299f7c6e9a05d9071c98e5d4fb9426dec0e57d7f58883a8e853 not found: ID does not exist" containerID="f14c009396fc0299f7c6e9a05d9071c98e5d4fb9426dec0e57d7f58883a8e853"
Sep 10 20:10:52.573403 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:52.573375    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14c009396fc0299f7c6e9a05d9071c98e5d4fb9426dec0e57d7f58883a8e853"} err="failed to get container status \"f14c009396fc0299f7c6e9a05d9071c98e5d4fb9426dec0e57d7f58883a8e853\": rpc error: code = NotFound desc = could not find container \"f14c009396fc0299f7c6e9a05d9071c98e5d4fb9426dec0e57d7f58883a8e853\": container with ID starting with f14c009396fc0299f7c6e9a05d9071c98e5d4fb9426dec0e57d7f58883a8e853 not found: ID does not exist"
Sep 10 20:10:52.592691 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:52.592663    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"]
Sep 10 20:10:52.595989 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:52.595965    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-39487c/test-model-39487c-kserve-696789c7b5-bbh46"]
Sep 10 20:10:53.404658 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:53.404624    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cca45dd-4408-4d5e-8307-59864450eac8" path="/var/lib/kubelet/pods/4cca45dd-4408-4d5e-8307-59864450eac8/volumes"
Sep 10 20:10:53.405009 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:53.404997    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8adcd939-74c3-4525-bd1d-4eaafbe9884c" path="/var/lib/kubelet/pods/8adcd939-74c3-4525-bd1d-4eaafbe9884c/volumes"
Sep 10 20:10:55.626548 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:55.626458    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:56.508193 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:56.508135    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:57.368751 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:57.368717    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:57.476583 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:57.476554    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:57.686144 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:57.686056    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:58.183654 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:58.183622    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:59.500078 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500044    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"]
Sep 10 20:10:59.500487 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500390    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f8aef37-376e-4b90-b249-fb62121e9953" containerName="maas-api"
Sep 10 20:10:59.500487 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500400    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f8aef37-376e-4b90-b249-fb62121e9953" containerName="maas-api"
Sep 10 20:10:59.500487 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500407    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cca45dd-4408-4d5e-8307-59864450eac8" containerName="main"
Sep 10 20:10:59.500487 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500412    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cca45dd-4408-4d5e-8307-59864450eac8" containerName="main"
Sep 10 20:10:59.500487 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500426    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8adcd939-74c3-4525-bd1d-4eaafbe9884c" containerName="main"
Sep 10 20:10:59.500487 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500432    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8adcd939-74c3-4525-bd1d-4eaafbe9884c" containerName="main"
Sep 10 20:10:59.500487 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500443    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="051f910f-86bf-49fa-93ed-9189bdc21f17" containerName="maas-api"
Sep 10 20:10:59.500487 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500448    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="051f910f-86bf-49fa-93ed-9189bdc21f17" containerName="maas-api"
Sep 10 20:10:59.500487 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500458    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d509851-9eae-4c49-8585-2d87c1de5bc5" containerName="maas-api"
Sep 10 20:10:59.500487 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500464    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d509851-9eae-4c49-8585-2d87c1de5bc5" containerName="maas-api"
Sep 10 20:10:59.500872 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500515    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="051f910f-86bf-49fa-93ed-9189bdc21f17" containerName="maas-api"
Sep 10 20:10:59.500872 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500521    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d509851-9eae-4c49-8585-2d87c1de5bc5" containerName="maas-api"
Sep 10 20:10:59.500872 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500542    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f8aef37-376e-4b90-b249-fb62121e9953" containerName="maas-api"
Sep 10 20:10:59.500872 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500549    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cca45dd-4408-4d5e-8307-59864450eac8" containerName="main"
Sep 10 20:10:59.500872 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.500556    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8adcd939-74c3-4525-bd1d-4eaafbe9884c" containerName="main"
Sep 10 20:10:59.504656 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.504640    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.507770 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.507745    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-842160\"/\"default-dockercfg-x88dg\""
Sep 10 20:10:59.508440 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.508420    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-842160\"/\"kube-root-ca.crt\""
Sep 10 20:10:59.508599 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.508478    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-842160\"/\"openshift-service-ca.crt\""
Sep 10 20:10:59.508684 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.508602    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-842160\"/\"noten-model-842160-kserve-self-signed-certs\""
Sep 10 20:10:59.519466 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.519436    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"]
Sep 10 20:10:59.622537 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.622503    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f046883e-999f-4b93-92de-880d4455320b-tls-certs\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.622714 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.622569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n8kz\" (UniqueName: \"kubernetes.io/projected/f046883e-999f-4b93-92de-880d4455320b-kube-api-access-6n8kz\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.622714 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.622608    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-model-cache\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.622714 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.622666    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-dshm\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.622824 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.622730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-tmp-dir\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.622824 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.622770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-home\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.642427 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.642397    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:10:59.724052 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.724017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-dshm\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.724234 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.724067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-tmp-dir\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.724234 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.724100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-home\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.724234 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.724167    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f046883e-999f-4b93-92de-880d4455320b-tls-certs\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.724234 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.724196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n8kz\" (UniqueName: \"kubernetes.io/projected/f046883e-999f-4b93-92de-880d4455320b-kube-api-access-6n8kz\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.724234 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.724219    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-model-cache\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.724629 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.724586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-tmp-dir\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.724629 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.724624    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-home\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.724820 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.724661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-model-cache\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.726463 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.726444    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-dshm\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.726803 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.726776    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f046883e-999f-4b93-92de-880d4455320b-tls-certs\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.736119 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.736087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n8kz\" (UniqueName: \"kubernetes.io/projected/f046883e-999f-4b93-92de-880d4455320b-kube-api-access-6n8kz\") pod \"noten-model-842160-kserve-7d8b579fbc-q6mzx\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") " pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:10:59.815003 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:10:59.814966    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:11:00.096136 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:00.096107    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"]
Sep 10 20:11:00.097503 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:11:00.097480    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf046883e_999f_4b93_92de_880d4455320b.slice/crio-3288a1baa1dff3a7bc73eaf5043ba74b123ac0f13e955a37f18a61059131c0a1 WatchSource:0}: Error finding container 3288a1baa1dff3a7bc73eaf5043ba74b123ac0f13e955a37f18a61059131c0a1: Status 404 returned error can't find the container with id 3288a1baa1dff3a7bc73eaf5043ba74b123ac0f13e955a37f18a61059131c0a1
Sep 10 20:11:00.610047 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:00.610012    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx" event={"ID":"f046883e-999f-4b93-92de-880d4455320b","Type":"ContainerStarted","Data":"9af8a8173084e85aae4a24f027dc97d995a99e05063691da9336b99044c8e7cc"}
Sep 10 20:11:00.610047 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:00.610054    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx" event={"ID":"f046883e-999f-4b93-92de-880d4455320b","Type":"ContainerStarted","Data":"3288a1baa1dff3a7bc73eaf5043ba74b123ac0f13e955a37f18a61059131c0a1"}
Sep 10 20:11:00.634384 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:00.634327    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx" podStartSLOduration=1.63431145 podStartE2EDuration="1.63431145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:11:00.632757317 +0000 UTC m=+1001.683285305" watchObservedRunningTime="2026-09-10 20:11:00.63431145 +0000 UTC m=+1001.684839514"
Sep 10 20:11:02.976634 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:02.976596    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:03.132452 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:03.132418    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:03.237115 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:03.237031    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:03.346987 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:03.346947    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:04.142696 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:04.142655    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:04.256553 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:04.256494    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:05.805981 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:05.805950    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:08.097731 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:08.097699    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:08.756329 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:08.756288    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"]
Sep 10 20:11:08.756538 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:08.756503    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q" podUID="0151370b-d749-4880-8c7b-9a50be65a2a6" containerName="maas-api" containerID="cri-o://6f52f7cfe4b46d28eacfaa861af86490653ddafbda0ab973d75b0bd1416ea09e" gracePeriod=30
Sep 10 20:11:08.791070 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:08.791040    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:08.912287 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:08.912244    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:09.013735 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:09.013654    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:09.136104 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:09.136073    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:09.234303 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:09.234268    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:09.516412 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:09.516377    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:09.815673 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:09.815641    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:11:09.815884 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:09.815713    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:11:09.828618 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:09.828585    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:11:10.007025 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.006994    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:11:10.122851 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.122763    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68bjb\" (UniqueName: \"kubernetes.io/projected/0151370b-d749-4880-8c7b-9a50be65a2a6-kube-api-access-68bjb\") pod \"0151370b-d749-4880-8c7b-9a50be65a2a6\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") "
Sep 10 20:11:10.122851 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.122820    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0151370b-d749-4880-8c7b-9a50be65a2a6-metrics-certs\") pod \"0151370b-d749-4880-8c7b-9a50be65a2a6\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") "
Sep 10 20:11:10.123090 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.122906    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0151370b-d749-4880-8c7b-9a50be65a2a6-openshift-service-ca\") pod \"0151370b-d749-4880-8c7b-9a50be65a2a6\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") "
Sep 10 20:11:10.123090 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.122922    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0151370b-d749-4880-8c7b-9a50be65a2a6-maas-api-tls\") pod \"0151370b-d749-4880-8c7b-9a50be65a2a6\" (UID: \"0151370b-d749-4880-8c7b-9a50be65a2a6\") "
Sep 10 20:11:10.123292 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.123265    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0151370b-d749-4880-8c7b-9a50be65a2a6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0151370b-d749-4880-8c7b-9a50be65a2a6" (UID: "0151370b-d749-4880-8c7b-9a50be65a2a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:11:10.125149 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.125119    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0151370b-d749-4880-8c7b-9a50be65a2a6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0151370b-d749-4880-8c7b-9a50be65a2a6" (UID: "0151370b-d749-4880-8c7b-9a50be65a2a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:11:10.125149 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.125141    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0151370b-d749-4880-8c7b-9a50be65a2a6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0151370b-d749-4880-8c7b-9a50be65a2a6" (UID: "0151370b-d749-4880-8c7b-9a50be65a2a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:11:10.125310 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.125192    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0151370b-d749-4880-8c7b-9a50be65a2a6-kube-api-access-68bjb" (OuterVolumeSpecName: "kube-api-access-68bjb") pod "0151370b-d749-4880-8c7b-9a50be65a2a6" (UID: "0151370b-d749-4880-8c7b-9a50be65a2a6"). InnerVolumeSpecName "kube-api-access-68bjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:11:10.224480 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.224445    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0151370b-d749-4880-8c7b-9a50be65a2a6-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:11:10.224480 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.224476    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0151370b-d749-4880-8c7b-9a50be65a2a6-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:11:10.224480 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.224487    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68bjb\" (UniqueName: \"kubernetes.io/projected/0151370b-d749-4880-8c7b-9a50be65a2a6-kube-api-access-68bjb\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:11:10.224964 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.224496    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0151370b-d749-4880-8c7b-9a50be65a2a6-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:11:10.663434 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.663398    2561 generic.go:356] "Generic (PLEG): container finished" podID="0151370b-d749-4880-8c7b-9a50be65a2a6" containerID="6f52f7cfe4b46d28eacfaa861af86490653ddafbda0ab973d75b0bd1416ea09e" exitCode=0
Sep 10 20:11:10.663646 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.663447    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"
Sep 10 20:11:10.663646 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.663488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q" event={"ID":"0151370b-d749-4880-8c7b-9a50be65a2a6","Type":"ContainerDied","Data":"6f52f7cfe4b46d28eacfaa861af86490653ddafbda0ab973d75b0bd1416ea09e"}
Sep 10 20:11:10.663646 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.663568    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q" event={"ID":"0151370b-d749-4880-8c7b-9a50be65a2a6","Type":"ContainerDied","Data":"11e8f3b0d2fb44754b2887a908bf99bec2160377e41786e34c963289ad1c315f"}
Sep 10 20:11:10.663646 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.663595    2561 scope.go:117] "RemoveContainer" containerID="6f52f7cfe4b46d28eacfaa861af86490653ddafbda0ab973d75b0bd1416ea09e"
Sep 10 20:11:10.674624 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.674606    2561 scope.go:117] "RemoveContainer" containerID="6f52f7cfe4b46d28eacfaa861af86490653ddafbda0ab973d75b0bd1416ea09e"
Sep 10 20:11:10.674916 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:11:10.674896    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f52f7cfe4b46d28eacfaa861af86490653ddafbda0ab973d75b0bd1416ea09e\": container with ID starting with 6f52f7cfe4b46d28eacfaa861af86490653ddafbda0ab973d75b0bd1416ea09e not found: ID does not exist" containerID="6f52f7cfe4b46d28eacfaa861af86490653ddafbda0ab973d75b0bd1416ea09e"
Sep 10 20:11:10.675024 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.674923    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f52f7cfe4b46d28eacfaa861af86490653ddafbda0ab973d75b0bd1416ea09e"} err="failed to get container status \"6f52f7cfe4b46d28eacfaa861af86490653ddafbda0ab973d75b0bd1416ea09e\": rpc error: code = NotFound desc = could not find container \"6f52f7cfe4b46d28eacfaa861af86490653ddafbda0ab973d75b0bd1416ea09e\": container with ID starting with 6f52f7cfe4b46d28eacfaa861af86490653ddafbda0ab973d75b0bd1416ea09e not found: ID does not exist"
Sep 10 20:11:10.677571 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.677547    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:11:10.690252 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.690222    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"]
Sep 10 20:11:10.701909 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:10.701882    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39487c-5978d4bdfd-5sj4q"]
Sep 10 20:11:11.403792 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:11.403757    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0151370b-d749-4880-8c7b-9a50be65a2a6" path="/var/lib/kubelet/pods/0151370b-d749-4880-8c7b-9a50be65a2a6/volumes"
Sep 10 20:11:12.363165 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:12.363133    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:12.519408 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:12.519372    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:13.125489 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:13.125457    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:15.657868 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:15.657832    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:15.756791 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:15.756737    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:15.940386 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:15.940296    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:16.500513 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:16.500481    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:16.849605 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:16.849569    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:16.943426 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:16.943388    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:19.824345 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:19.824311    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:19.929395 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:19.929361    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:20.065363 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:20.065336    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:20.451074 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:20.451037    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:20.609155 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:20.609118    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:21.159488 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.159454    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"]
Sep 10 20:11:21.159899 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.159810    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0151370b-d749-4880-8c7b-9a50be65a2a6" containerName="maas-api"
Sep 10 20:11:21.159899 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.159822    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0151370b-d749-4880-8c7b-9a50be65a2a6" containerName="maas-api"
Sep 10 20:11:21.159899 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.159890    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0151370b-d749-4880-8c7b-9a50be65a2a6" containerName="maas-api"
Sep 10 20:11:21.163907 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.163889    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:21.167377 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.167356    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-591d7c-serving-cert\""
Sep 10 20:11:21.167504 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.167479    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:21.185226 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.185195    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"]
Sep 10 20:11:21.285092 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.285061    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:21.333754 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.333717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59419042-05a6-477e-ae6b-f6eab5eda4dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:21.333955 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.333783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59419042-05a6-477e-ae6b-f6eab5eda4dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:21.333955 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.333863    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59419042-05a6-477e-ae6b-f6eab5eda4dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:21.334053 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.333962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wmp6\" (UniqueName: \"kubernetes.io/projected/59419042-05a6-477e-ae6b-f6eab5eda4dc-kube-api-access-2wmp6\") pod \"maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:21.412314 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.412228    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:21.435478 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.435444    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59419042-05a6-477e-ae6b-f6eab5eda4dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:21.435613 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.435496    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59419042-05a6-477e-ae6b-f6eab5eda4dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:21.435613 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.435579    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wmp6\" (UniqueName: \"kubernetes.io/projected/59419042-05a6-477e-ae6b-f6eab5eda4dc-kube-api-access-2wmp6\") pod \"maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:21.435727 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.435636    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59419042-05a6-477e-ae6b-f6eab5eda4dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:21.436178 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.436149    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59419042-05a6-477e-ae6b-f6eab5eda4dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:21.438162 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.438135    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59419042-05a6-477e-ae6b-f6eab5eda4dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:21.438267 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.438161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59419042-05a6-477e-ae6b-f6eab5eda4dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:21.446876 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.446849    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wmp6\" (UniqueName: \"kubernetes.io/projected/59419042-05a6-477e-ae6b-f6eab5eda4dc-kube-api-access-2wmp6\") pod \"maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:21.474686 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.474652    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:21.561181 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.561132    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:21.768199 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.768168    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"]
Sep 10 20:11:21.771163 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:11:21.771137    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59419042_05a6_477e_ae6b_f6eab5eda4dc.slice/crio-328a0a4e7af94868d8dd45dfb0561b8b285a37b4336de86389ede2239e28a9b7 WatchSource:0}: Error finding container 328a0a4e7af94868d8dd45dfb0561b8b285a37b4336de86389ede2239e28a9b7: Status 404 returned error can't find the container with id 328a0a4e7af94868d8dd45dfb0561b8b285a37b4336de86389ede2239e28a9b7
Sep 10 20:11:21.772601 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.772584    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:11:21.799806 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.799757    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:21.902839 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:21.902803    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:22.727690 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:22.727656    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps" event={"ID":"59419042-05a6-477e-ae6b-f6eab5eda4dc","Type":"ContainerStarted","Data":"90bcea8240a080a34a02431f936b89e7caa947a63a4bf9676a4e3a96f6d7a72b"}
Sep 10 20:11:22.728079 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:22.727713    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:22.728079 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:22.727734    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps" event={"ID":"59419042-05a6-477e-ae6b-f6eab5eda4dc","Type":"ContainerStarted","Data":"328a0a4e7af94868d8dd45dfb0561b8b285a37b4336de86389ede2239e28a9b7"}
Sep 10 20:11:22.750483 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:22.750432    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps" podStartSLOduration=1.348290268 podStartE2EDuration="1.750416182s" totalImagesPullingTime="402.125914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:11:22.748010596 +0000 UTC m=+1023.798538586" watchObservedRunningTime="2026-09-10 20:11:22.750416182 +0000 UTC m=+1023.800944170"
Sep 10 20:11:24.050363 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:24.050329    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:24.148462 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:24.148428    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:25.196650 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.196616    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"]
Sep 10 20:11:25.199930 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.199909    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:25.203497 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.203474    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2017ae11-serving-cert\""
Sep 10 20:11:25.218582 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.218554    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"]
Sep 10 20:11:25.253323 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.253285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:25.353176 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.353134    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:25.373954 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.373924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-metrics-certs\") pod \"maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:25.373954 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.373952    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2tpb\" (UniqueName: \"kubernetes.io/projected/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-kube-api-access-l2tpb\") pod \"maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:25.374164 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.373978    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-openshift-service-ca\") pod \"maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:25.374164 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.374086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-maas-api-tls\") pod \"maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:25.474870 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.474774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-metrics-certs\") pod \"maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:25.474870 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.474815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2tpb\" (UniqueName: \"kubernetes.io/projected/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-kube-api-access-l2tpb\") pod \"maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:25.474870 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.474852    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-openshift-service-ca\") pod \"maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:25.475178 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.474924    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-maas-api-tls\") pod \"maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:25.475178 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:11:25.475066    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-serving-cert: secret "maas-api-e2e-mt-2017ae11-serving-cert" not found
Sep 10 20:11:25.475178 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:11:25.475141    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-maas-api-tls podName:ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a nodeName:}" failed. No retries permitted until 2026-09-10 20:11:25.975119374 +0000 UTC m=+1027.025647356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-maas-api-tls") pod "maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv" (UID: "ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a") : secret "maas-api-e2e-mt-2017ae11-serving-cert" not found
Sep 10 20:11:25.475592 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.475566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-openshift-service-ca\") pod \"maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:25.477561 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.477540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-metrics-certs\") pod \"maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:25.485131 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.485111    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2tpb\" (UniqueName: \"kubernetes.io/projected/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-kube-api-access-l2tpb\") pod \"maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:25.646491 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.646460    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:25.927933 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.927898    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:25.987243 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.987212    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-maas-api-tls\") pod \"maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:25.989734 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:25.989714    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-maas-api-tls\") pod \"maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:26.024124 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:26.024087    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:26.111140 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:26.111100    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:26.399383 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:26.399347    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"]
Sep 10 20:11:26.412510 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:11:26.412482    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee78e58e_e7f7_479d_8b1a_eaf287fd7a8a.slice/crio-b710db9e4c19dc133f770fb4ca876ce8848d92c4df69c291e3a009d45e13d7ee WatchSource:0}: Error finding container b710db9e4c19dc133f770fb4ca876ce8848d92c4df69c291e3a009d45e13d7ee: Status 404 returned error can't find the container with id b710db9e4c19dc133f770fb4ca876ce8848d92c4df69c291e3a009d45e13d7ee
Sep 10 20:11:26.750110 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:26.750025    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv" event={"ID":"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a","Type":"ContainerStarted","Data":"b710db9e4c19dc133f770fb4ca876ce8848d92c4df69c291e3a009d45e13d7ee"}
Sep 10 20:11:27.756806 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:27.756760    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv" event={"ID":"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a","Type":"ContainerStarted","Data":"bfd71b15b4e3dd902d70dc0d42e5bb23d0363c6cd3e332ebc1a3a3cb53cc16c9"}
Sep 10 20:11:27.757308 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:27.756898    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:27.781298 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:27.781246    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv" podStartSLOduration=2.16422297 podStartE2EDuration="2.781229934s" totalImagesPullingTime="617.006964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:11:27.778443448 +0000 UTC m=+1028.828971438" watchObservedRunningTime="2026-09-10 20:11:27.781229934 +0000 UTC m=+1028.831757923"
Sep 10 20:11:28.737924 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:28.737897    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:11:32.719029 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:32.718992    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:33.768470 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:33.768439    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:11:34.334199 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:34.334162    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:34.546452 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:34.546399    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:34.712772 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:34.712674    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:39.627326 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:39.627297    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"]
Sep 10 20:11:39.722098 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:39.722065    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:42.322431 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:42.322390    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:44.820662 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:44.820627    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:45.175204 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:45.175104    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:45.581409 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:45.581376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:45.776249 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:45.776218    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:46.751899 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:46.751858    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:48.011708 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:48.011674    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:48.376009 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:48.375971    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:48.494702 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:48.494664    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:49.002663 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.002626    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:49.504514 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.504477    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"]
Sep 10 20:11:49.507991 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.507967    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:49.511143 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:11:49.511115    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-e2e-shared-b-w3-6550b4-serving-cert\" is forbidden: User \"system:node:ip-10-0-140-111.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-140-111.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-6550b4-serving-cert\"" type="*v1.Secret"
Sep 10 20:11:49.511285 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:11:49.511235    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb\" is forbidden: User \"system:node:ip-10-0-140-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-140-111.ec2.internal' and this object" podUID="21e127b8-bdca-4dbc-848e-afb20ad8fb00" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:49.533932 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.533903    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"]
Sep 10 20:11:49.573115 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.573082    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:49.592254 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.592223    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e127b8-bdca-4dbc-848e-afb20ad8fb00-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:49.592411 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.592281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e127b8-bdca-4dbc-848e-afb20ad8fb00-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:49.592411 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.592326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsxbt\" (UniqueName: \"kubernetes.io/projected/21e127b8-bdca-4dbc-848e-afb20ad8fb00-kube-api-access-zsxbt\") pod \"maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:49.592411 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.592400    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e127b8-bdca-4dbc-848e-afb20ad8fb00-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:49.692944 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.692902    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e127b8-bdca-4dbc-848e-afb20ad8fb00-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:49.693125 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.692987    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e127b8-bdca-4dbc-848e-afb20ad8fb00-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:49.693125 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.693023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsxbt\" (UniqueName: \"kubernetes.io/projected/21e127b8-bdca-4dbc-848e-afb20ad8fb00-kube-api-access-zsxbt\") pod \"maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:49.693260 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.693126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e127b8-bdca-4dbc-848e-afb20ad8fb00-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:49.693867 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.693839    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e127b8-bdca-4dbc-848e-afb20ad8fb00-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:49.696094 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.696072    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e127b8-bdca-4dbc-848e-afb20ad8fb00-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:49.707117 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.707087    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:49.708052 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.708032    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsxbt\" (UniqueName: \"kubernetes.io/projected/21e127b8-bdca-4dbc-848e-afb20ad8fb00-kube-api-access-zsxbt\") pod \"maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:49.809516 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:49.809481    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:50.147226 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:50.147136    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:50.250912 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:50.250880    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:50.332786 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:50.332753    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-6550b4-serving-cert\""
Sep 10 20:11:50.336155 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:50.336129    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e127b8-bdca-4dbc-848e-afb20ad8fb00-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:50.419425 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:50.419336    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:50.532836 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:50.532797    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:50.719926 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:50.719899    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"]
Sep 10 20:11:50.721234 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:11:50.721209    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e127b8_bdca_4dbc_848e_afb20ad8fb00.slice/crio-d2162bd29cf8cd1be17acab775a3a25def7d7342a832c73c1a24ddc36160c264 WatchSource:0}: Error finding container d2162bd29cf8cd1be17acab775a3a25def7d7342a832c73c1a24ddc36160c264: Status 404 returned error can't find the container with id d2162bd29cf8cd1be17acab775a3a25def7d7342a832c73c1a24ddc36160c264
Sep 10 20:11:50.836365 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:50.836335    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:50.882325 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:50.882291    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb" event={"ID":"21e127b8-bdca-4dbc-848e-afb20ad8fb00","Type":"ContainerStarted","Data":"d2162bd29cf8cd1be17acab775a3a25def7d7342a832c73c1a24ddc36160c264"}
Sep 10 20:11:51.889139 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:51.889103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb" event={"ID":"21e127b8-bdca-4dbc-848e-afb20ad8fb00","Type":"ContainerStarted","Data":"64220f7c669e155b14799de79f985703dc16ac19652220a5437d9030a9b05fae"}
Sep 10 20:11:51.889605 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:51.889236    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:11:51.909435 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:51.909386    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb" podStartSLOduration=2.521046899 podStartE2EDuration="2.909370409s" totalImagesPullingTime="388.32351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:11:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:11:51.908124362 +0000 UTC m=+1052.958652341" watchObservedRunningTime="2026-09-10 20:11:51.909370409 +0000 UTC m=+1052.959898453"
Sep 10 20:11:54.648004 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:54.647962    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx" podUID="f046883e-999f-4b93-92de-880d4455320b" containerName="main" containerID="cri-o://9af8a8173084e85aae4a24f027dc97d995a99e05063691da9336b99044c8e7cc" gracePeriod=45
Sep 10 20:11:54.904074 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:54.904011    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:11:54.906473 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:54.906447    2561 generic.go:356] "Generic (PLEG): container finished" podID="f046883e-999f-4b93-92de-880d4455320b" containerID="9af8a8173084e85aae4a24f027dc97d995a99e05063691da9336b99044c8e7cc" exitCode=0
Sep 10 20:11:54.906647 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:54.906485    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"
Sep 10 20:11:54.906647 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:54.906543    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx" event={"ID":"f046883e-999f-4b93-92de-880d4455320b","Type":"ContainerDied","Data":"9af8a8173084e85aae4a24f027dc97d995a99e05063691da9336b99044c8e7cc"}
Sep 10 20:11:54.906647 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:54.906593    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx" event={"ID":"f046883e-999f-4b93-92de-880d4455320b","Type":"ContainerDied","Data":"3288a1baa1dff3a7bc73eaf5043ba74b123ac0f13e955a37f18a61059131c0a1"}
Sep 10 20:11:54.906647 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:54.906615    2561 scope.go:117] "RemoveContainer" containerID="9af8a8173084e85aae4a24f027dc97d995a99e05063691da9336b99044c8e7cc"
Sep 10 20:11:54.915816 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:54.915797    2561 scope.go:117] "RemoveContainer" containerID="9af8a8173084e85aae4a24f027dc97d995a99e05063691da9336b99044c8e7cc"
Sep 10 20:11:54.916076 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:11:54.916060    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af8a8173084e85aae4a24f027dc97d995a99e05063691da9336b99044c8e7cc\": container with ID starting with 9af8a8173084e85aae4a24f027dc97d995a99e05063691da9336b99044c8e7cc not found: ID does not exist" containerID="9af8a8173084e85aae4a24f027dc97d995a99e05063691da9336b99044c8e7cc"
Sep 10 20:11:54.916122 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:54.916086    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af8a8173084e85aae4a24f027dc97d995a99e05063691da9336b99044c8e7cc"} err="failed to get container status \"9af8a8173084e85aae4a24f027dc97d995a99e05063691da9336b99044c8e7cc\": rpc error: code = NotFound desc = could not find container \"9af8a8173084e85aae4a24f027dc97d995a99e05063691da9336b99044c8e7cc\": container with ID starting with 9af8a8173084e85aae4a24f027dc97d995a99e05063691da9336b99044c8e7cc not found: ID does not exist"
Sep 10 20:11:55.044722 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.044689    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n8kz\" (UniqueName: \"kubernetes.io/projected/f046883e-999f-4b93-92de-880d4455320b-kube-api-access-6n8kz\") pod \"f046883e-999f-4b93-92de-880d4455320b\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") "
Sep 10 20:11:55.044946 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.044733    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-dshm\") pod \"f046883e-999f-4b93-92de-880d4455320b\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") "
Sep 10 20:11:55.044946 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.044788    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f046883e-999f-4b93-92de-880d4455320b-tls-certs\") pod \"f046883e-999f-4b93-92de-880d4455320b\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") "
Sep 10 20:11:55.044946 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.044824    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-model-cache\") pod \"f046883e-999f-4b93-92de-880d4455320b\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") "
Sep 10 20:11:55.044946 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.044856    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-tmp-dir\") pod \"f046883e-999f-4b93-92de-880d4455320b\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") "
Sep 10 20:11:55.044946 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.044904    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-home\") pod \"f046883e-999f-4b93-92de-880d4455320b\" (UID: \"f046883e-999f-4b93-92de-880d4455320b\") "
Sep 10 20:11:55.045223 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.045145    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f046883e-999f-4b93-92de-880d4455320b" (UID: "f046883e-999f-4b93-92de-880d4455320b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:11:55.045223 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.045149    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f046883e-999f-4b93-92de-880d4455320b" (UID: "f046883e-999f-4b93-92de-880d4455320b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:11:55.045223 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.045206    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-home" (OuterVolumeSpecName: "home") pod "f046883e-999f-4b93-92de-880d4455320b" (UID: "f046883e-999f-4b93-92de-880d4455320b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:11:55.047176 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.047147    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-dshm" (OuterVolumeSpecName: "dshm") pod "f046883e-999f-4b93-92de-880d4455320b" (UID: "f046883e-999f-4b93-92de-880d4455320b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:11:55.047271 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.047197    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f046883e-999f-4b93-92de-880d4455320b-kube-api-access-6n8kz" (OuterVolumeSpecName: "kube-api-access-6n8kz") pod "f046883e-999f-4b93-92de-880d4455320b" (UID: "f046883e-999f-4b93-92de-880d4455320b"). InnerVolumeSpecName "kube-api-access-6n8kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:11:55.047271 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.047208    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f046883e-999f-4b93-92de-880d4455320b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f046883e-999f-4b93-92de-880d4455320b" (UID: "f046883e-999f-4b93-92de-880d4455320b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:11:55.145704 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.145673    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:11:55.145704 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.145703    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n8kz\" (UniqueName: \"kubernetes.io/projected/f046883e-999f-4b93-92de-880d4455320b-kube-api-access-6n8kz\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:11:55.145704 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.145714    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:11:55.145938 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.145723    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f046883e-999f-4b93-92de-880d4455320b-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:11:55.145938 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.145731    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:11:55.145938 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.145738    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f046883e-999f-4b93-92de-880d4455320b-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:11:55.243515 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.243473    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"]
Sep 10 20:11:55.252403 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.252372    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-842160/noten-model-842160-kserve-7d8b579fbc-q6mzx"]
Sep 10 20:11:55.404951 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:55.404916    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f046883e-999f-4b93-92de-880d4455320b" path="/var/lib/kubelet/pods/f046883e-999f-4b93-92de-880d4455320b/volumes"
Sep 10 20:11:56.950908 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:56.950881    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:57.044503 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:57.044469    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:57.459111 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:57.459073    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:11:57.900446 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:11:57.900414    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:12:08.426696 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.426641    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"]
Sep 10 20:12:08.427157 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.426905    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv" podUID="ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a" containerName="maas-api" containerID="cri-o://bfd71b15b4e3dd902d70dc0d42e5bb23d0363c6cd3e332ebc1a3a3cb53cc16c9" gracePeriod=30
Sep 10 20:12:08.491148 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.491115    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:08.688077 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.688020    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:12:08.768706 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.768676    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-metrics-certs\") pod \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") "
Sep 10 20:12:08.768875 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.768715    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-maas-api-tls\") pod \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") "
Sep 10 20:12:08.768875 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.768745    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2tpb\" (UniqueName: \"kubernetes.io/projected/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-kube-api-access-l2tpb\") pod \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") "
Sep 10 20:12:08.768875 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.768801    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-openshift-service-ca\") pod \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\" (UID: \"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a\") "
Sep 10 20:12:08.769302 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.769245    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a" (UID: "ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:12:08.771182 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.771153    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-kube-api-access-l2tpb" (OuterVolumeSpecName: "kube-api-access-l2tpb") pod "ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a" (UID: "ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a"). InnerVolumeSpecName "kube-api-access-l2tpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:12:08.771288 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.771178    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a" (UID: "ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:12:08.771288 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.771191    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a" (UID: "ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:12:08.870290 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.870253    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:12:08.870290 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.870285    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:12:08.870290 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.870299    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:12:08.870571 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.870310    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2tpb\" (UniqueName: \"kubernetes.io/projected/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a-kube-api-access-l2tpb\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:12:08.983185 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.983094    2561 generic.go:356] "Generic (PLEG): container finished" podID="ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a" containerID="bfd71b15b4e3dd902d70dc0d42e5bb23d0363c6cd3e332ebc1a3a3cb53cc16c9" exitCode=0
Sep 10 20:12:08.983185 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.983135    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"
Sep 10 20:12:08.983397 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.983183    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv" event={"ID":"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a","Type":"ContainerDied","Data":"bfd71b15b4e3dd902d70dc0d42e5bb23d0363c6cd3e332ebc1a3a3cb53cc16c9"}
Sep 10 20:12:08.983397 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.983231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv" event={"ID":"ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a","Type":"ContainerDied","Data":"b710db9e4c19dc133f770fb4ca876ce8848d92c4df69c291e3a009d45e13d7ee"}
Sep 10 20:12:08.983397 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.983255    2561 scope.go:117] "RemoveContainer" containerID="bfd71b15b4e3dd902d70dc0d42e5bb23d0363c6cd3e332ebc1a3a3cb53cc16c9"
Sep 10 20:12:08.992860 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.992843    2561 scope.go:117] "RemoveContainer" containerID="bfd71b15b4e3dd902d70dc0d42e5bb23d0363c6cd3e332ebc1a3a3cb53cc16c9"
Sep 10 20:12:08.993133 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:12:08.993117    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd71b15b4e3dd902d70dc0d42e5bb23d0363c6cd3e332ebc1a3a3cb53cc16c9\": container with ID starting with bfd71b15b4e3dd902d70dc0d42e5bb23d0363c6cd3e332ebc1a3a3cb53cc16c9 not found: ID does not exist" containerID="bfd71b15b4e3dd902d70dc0d42e5bb23d0363c6cd3e332ebc1a3a3cb53cc16c9"
Sep 10 20:12:08.993187 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:08.993143    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd71b15b4e3dd902d70dc0d42e5bb23d0363c6cd3e332ebc1a3a3cb53cc16c9"} err="failed to get container status \"bfd71b15b4e3dd902d70dc0d42e5bb23d0363c6cd3e332ebc1a3a3cb53cc16c9\": rpc error: code = NotFound desc = could not find container \"bfd71b15b4e3dd902d70dc0d42e5bb23d0363c6cd3e332ebc1a3a3cb53cc16c9\": container with ID starting with bfd71b15b4e3dd902d70dc0d42e5bb23d0363c6cd3e332ebc1a3a3cb53cc16c9 not found: ID does not exist"
Sep 10 20:12:09.058181 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:09.058149    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"]
Sep 10 20:12:09.061064 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:09.061040    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2017ae11-8587b957f7-s7vgv"]
Sep 10 20:12:09.130770 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:09.130740    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:09.280065 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:09.280021    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:09.409600 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:09.405006    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a" path="/var/lib/kubelet/pods/ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a/volumes"
Sep 10 20:12:09.559979 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:09.559885    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:11.180661 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:11.180584    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:13.299340 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:13.299306    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:16.719211 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:16.719170    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:16.834816 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:16.834781    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:17.259957 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:17.259922    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:17.391463 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:17.391431    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:19.462076 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:19.462040    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:19.647311 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:19.647282    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:26.039036 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:26.039000    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:26.303479 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:26.303396    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:26.500731 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:26.500694    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:30.064437 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.064404    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"]
Sep 10 20:12:30.064854 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.064754    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a" containerName="maas-api"
Sep 10 20:12:30.064854 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.064768    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a" containerName="maas-api"
Sep 10 20:12:30.064854 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.064783    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f046883e-999f-4b93-92de-880d4455320b" containerName="main"
Sep 10 20:12:30.064854 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.064788    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f046883e-999f-4b93-92de-880d4455320b" containerName="main"
Sep 10 20:12:30.064854 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.064848    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee78e58e-e7f7-479d-8b1a-eaf287fd7a8a" containerName="maas-api"
Sep 10 20:12:30.065045 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.064860    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f046883e-999f-4b93-92de-880d4455320b" containerName="main"
Sep 10 20:12:30.069061 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.069041    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.072691 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.072663    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-591d7c\"/\"default-dockercfg-fkpb5\""
Sep 10 20:12:30.072835 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.072665    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-591d7c\"/\"openshift-service-ca.crt\""
Sep 10 20:12:30.072835 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.072665    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-591d7c\"/\"kube-root-ca.crt\""
Sep 10 20:12:30.072835 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.072722    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-591d7c\"/\"auth-test-model-591d7c-kserve-self-signed-certs\""
Sep 10 20:12:30.080047 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.080020    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"]
Sep 10 20:12:30.101084 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.101012    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:30.165417 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.165373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-tmp-dir\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.165624 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.165427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-model-cache\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.165624 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.165516    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-dshm\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.165624 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.165589    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22273f6-0690-49e9-82ad-931e9aa1659f-tls-certs\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.165624 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.165615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-home\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.165803 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.165654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p4dp\" (UniqueName: \"kubernetes.io/projected/a22273f6-0690-49e9-82ad-931e9aa1659f-kube-api-access-9p4dp\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.266425 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.266380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22273f6-0690-49e9-82ad-931e9aa1659f-tls-certs\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.266684 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.266433    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-home\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.266684 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.266474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p4dp\" (UniqueName: \"kubernetes.io/projected/a22273f6-0690-49e9-82ad-931e9aa1659f-kube-api-access-9p4dp\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.266684 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.266560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-tmp-dir\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.266684 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.266599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-model-cache\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.266684 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.266649    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-dshm\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.266978 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.266953    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-home\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.267092 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.266983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-tmp-dir\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.267092 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.267022    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-model-cache\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.269023 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.269002    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-dshm\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.269122 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.269085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22273f6-0690-49e9-82ad-931e9aa1659f-tls-certs\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.276586 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.276565    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p4dp\" (UniqueName: \"kubernetes.io/projected/a22273f6-0690-49e9-82ad-931e9aa1659f-kube-api-access-9p4dp\") pod \"auth-test-model-591d7c-kserve-58f7b556c-c8kfc\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.378314 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.378210    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:30.668319 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:30.668293    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"]
Sep 10 20:12:30.670468 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:12:30.670441    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda22273f6_0690_49e9_82ad_931e9aa1659f.slice/crio-7a6e8bea49b14336bbe53f8018dcfaf9ba798e13f6e6a42d4abe7f9131702180 WatchSource:0}: Error finding container 7a6e8bea49b14336bbe53f8018dcfaf9ba798e13f6e6a42d4abe7f9131702180: Status 404 returned error can't find the container with id 7a6e8bea49b14336bbe53f8018dcfaf9ba798e13f6e6a42d4abe7f9131702180
Sep 10 20:12:31.101667 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:31.101629    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc" event={"ID":"a22273f6-0690-49e9-82ad-931e9aa1659f","Type":"ContainerStarted","Data":"81eb365b27f33270c5cbf7a5c29ee5f4a8f2eb05532bdae81e895bf224e65b58"}
Sep 10 20:12:31.102052 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:31.101677    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc" event={"ID":"a22273f6-0690-49e9-82ad-931e9aa1659f","Type":"ContainerStarted","Data":"7a6e8bea49b14336bbe53f8018dcfaf9ba798e13f6e6a42d4abe7f9131702180"}
Sep 10 20:12:31.131864 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:31.131807    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc" podStartSLOduration=1.1317887070000001 podStartE2EDuration="1.131788707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:12:31.128145672 +0000 UTC m=+1092.178673661" watchObservedRunningTime="2026-09-10 20:12:31.131788707 +0000 UTC m=+1092.182316696"
Sep 10 20:12:33.375580 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:33.375519    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:33.485330 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:33.485285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:34.361489 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:34.361460    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:34.708814 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:34.708730    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:34.944597 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:34.944563    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:38.398479 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:38.398447    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:38.486626 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:38.486590    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:39.605091 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.605049    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"]
Sep 10 20:12:39.609138 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.609111    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:39.612361 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.612334    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-214b09-serving-cert\""
Sep 10 20:12:39.622126 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.622099    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"]
Sep 10 20:12:39.650112 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.650077    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:39.650294 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.650131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:39.650294 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.650209    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq29m\" (UniqueName: \"kubernetes.io/projected/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-kube-api-access-tq29m\") pod \"maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:39.650294 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.650281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:39.661092 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.661056    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:39.751450 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.751409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:39.751694 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.751461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:39.751694 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.751484    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq29m\" (UniqueName: \"kubernetes.io/projected/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-kube-api-access-tq29m\") pod \"maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:39.751694 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.751570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:39.751694 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:12:39.751642    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-serving-cert: secret "maas-api-e2e-shared-a-w2-214b09-serving-cert" not found
Sep 10 20:12:39.751904 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:12:39.751736    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-maas-api-tls podName:b0bae2ce-9f7d-49c5-bb89-691f723e34d1 nodeName:}" failed. No retries permitted until 2026-09-10 20:12:40.251715222 +0000 UTC m=+1101.302243210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-maas-api-tls") pod "maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr" (UID: "b0bae2ce-9f7d-49c5-bb89-691f723e34d1") : secret "maas-api-e2e-shared-a-w2-214b09-serving-cert" not found
Sep 10 20:12:39.752198 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.752182    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:39.754146 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.754123    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:39.761264 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.761236    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq29m\" (UniqueName: \"kubernetes.io/projected/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-kube-api-access-tq29m\") pod \"maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:39.761559 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.761510    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:39.981923 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:39.981823    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:40.257047 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:40.256944    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:40.259707 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:40.259684    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:40.266575 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:40.266520    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:40.372018 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:40.371982    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:12:40.378791 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:40.378763    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:40.378892 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:40.378851    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:40.391328 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:40.391300    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:40.522013 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:40.521985    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:40.800230 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:40.800068    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"]
Sep 10 20:12:40.802736 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:12:40.802707    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bae2ce_9f7d_49c5_bb89_691f723e34d1.slice/crio-d836f11c195a198c50a0edf58dc7816b977f99830f76de8aee09a8299154d0a6 WatchSource:0}: Error finding container d836f11c195a198c50a0edf58dc7816b977f99830f76de8aee09a8299154d0a6: Status 404 returned error can't find the container with id d836f11c195a198c50a0edf58dc7816b977f99830f76de8aee09a8299154d0a6
Sep 10 20:12:41.157133 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:41.157092    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr" event={"ID":"b0bae2ce-9f7d-49c5-bb89-691f723e34d1","Type":"ContainerStarted","Data":"d836f11c195a198c50a0edf58dc7816b977f99830f76de8aee09a8299154d0a6"}
Sep 10 20:12:41.168647 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:41.168607    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:12:42.163902 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:42.163867    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr" event={"ID":"b0bae2ce-9f7d-49c5-bb89-691f723e34d1","Type":"ContainerStarted","Data":"c7a5aa173660fd4c3bab923de7cb839587183bd8e00c0d582cadefb3e1b9ec1a"}
Sep 10 20:12:42.164289 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:42.164012    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:12:42.181415 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:42.181354    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr" podStartSLOduration=2.453049281 podStartE2EDuration="3.181325355s" totalImagesPullingTime="728.276074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:12:42.180712577 +0000 UTC m=+1103.231240566" watchObservedRunningTime="2026-09-10 20:12:42.181325355 +0000 UTC m=+1103.231853345"
Sep 10 20:12:48.175192 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:12:48.175165    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:13:03.397965 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:03.397926    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:03.735507 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:03.735419    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:04.099907 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:04.099869    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:04.328444 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:04.328407    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:07.686384 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:07.686344    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:07.806350 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:07.806314    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:08.945336 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:08.945304    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"]
Sep 10 20:13:08.949021 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:08.949002    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:08.952058 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:08.952035    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-6ef723-serving-cert\""
Sep 10 20:13:08.977585 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:08.977554    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"]
Sep 10 20:13:09.055868 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.055827    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:09.132320 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.132284    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:09.132320 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.132322    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:09.132557 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.132343    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqh5r\" (UniqueName: \"kubernetes.io/projected/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-kube-api-access-mqh5r\") pod \"maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:09.132557 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.132469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:09.189311 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.189277    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:09.233499 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.233402    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:09.233706 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.233512    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:09.233706 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.233575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:09.233706 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.233599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqh5r\" (UniqueName: \"kubernetes.io/projected/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-kube-api-access-mqh5r\") pod \"maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:09.234216 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.234186    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:09.236161 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.236142    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:09.236242 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.236143    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:09.249944 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.249909    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqh5r\" (UniqueName: \"kubernetes.io/projected/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-kube-api-access-mqh5r\") pod \"maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:09.259902 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.259881    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:09.407795 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.407770    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:09.546083 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.546056    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"]
Sep 10 20:13:09.547374 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:13:09.547350    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06eb9d90_f48b_4ea7_a353_bb274e0e43aa.slice/crio-f2691b211287c86f49e852b7c83452a08f9e101a710ea44d2972732cec833abb WatchSource:0}: Error finding container f2691b211287c86f49e852b7c83452a08f9e101a710ea44d2972732cec833abb: Status 404 returned error can't find the container with id f2691b211287c86f49e852b7c83452a08f9e101a710ea44d2972732cec833abb
Sep 10 20:13:09.790139 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.790103    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:09.898657 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:09.898566    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:10.312866 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:10.312835    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg" event={"ID":"06eb9d90-f48b-4ea7-a353-bb274e0e43aa","Type":"ContainerStarted","Data":"30fe97bfdaf1a3edd1df6ca7f28a6b26c453d9e3afbc67423520b76d466fc0d9"}
Sep 10 20:13:10.313268 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:10.312907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg" event={"ID":"06eb9d90-f48b-4ea7-a353-bb274e0e43aa","Type":"ContainerStarted","Data":"f2691b211287c86f49e852b7c83452a08f9e101a710ea44d2972732cec833abb"}
Sep 10 20:13:10.313268 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:10.312951    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:10.337277 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:10.337216    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg" podStartSLOduration=1.888428385 podStartE2EDuration="2.33719916s" totalImagesPullingTime="448.770775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:13:10.335207847 +0000 UTC m=+1131.385735837" watchObservedRunningTime="2026-09-10 20:13:10.33719916 +0000 UTC m=+1131.387727148"
Sep 10 20:13:14.240732 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:14.240694    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:14.341461 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:14.341422    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:16.324208 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:16.324173    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:13:29.242185 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.242148    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"]
Sep 10 20:13:29.245635 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.245612    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.248426 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.248392    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6550b4\"/\"kube-root-ca.crt\""
Sep 10 20:13:29.249376 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.249349    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6550b4\"/\"default-dockercfg-fnx6v\""
Sep 10 20:13:29.249684 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.249661    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6550b4\"/\"auth-test-model-6550b4-kserve-self-signed-certs\""
Sep 10 20:13:29.249800 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.249694    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6550b4\"/\"openshift-service-ca.crt\""
Sep 10 20:13:29.257547 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.257509    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"]
Sep 10 20:13:29.292266 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.292236    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:29.306166 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.306138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bbbx\" (UniqueName: \"kubernetes.io/projected/54c01746-d707-49ff-829f-0cfc1cdc03bb-kube-api-access-7bbbx\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.306331 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.306207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-model-cache\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.306331 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.306295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-dshm\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.306457 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.306344    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c01746-d707-49ff-829f-0cfc1cdc03bb-tls-certs\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.306457 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.306398    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-home\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.306457 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.306421    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-tmp-dir\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.407784 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.407749    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-model-cache\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.407949 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.407797    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-dshm\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.407949 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.407920    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c01746-d707-49ff-829f-0cfc1cdc03bb-tls-certs\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.408025 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.407976    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-home\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.408025 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.407999    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-tmp-dir\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.408103 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.408041    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bbbx\" (UniqueName: \"kubernetes.io/projected/54c01746-d707-49ff-829f-0cfc1cdc03bb-kube-api-access-7bbbx\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.408287 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.408263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-model-cache\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.408406 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.408299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-tmp-dir\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.408406 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.408330    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-home\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.410112 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.410085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-dshm\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.410431 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.410415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c01746-d707-49ff-829f-0cfc1cdc03bb-tls-certs\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.417328 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.417300    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bbbx\" (UniqueName: \"kubernetes.io/projected/54c01746-d707-49ff-829f-0cfc1cdc03bb-kube-api-access-7bbbx\") pod \"auth-test-model-6550b4-kserve-76d494c88c-4hh5c\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.558120 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.558077    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:29.835605 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:29.835569    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"]
Sep 10 20:13:29.837276 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:13:29.837250    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54c01746_d707_49ff_829f_0cfc1cdc03bb.slice/crio-10db60deed7a7205ca87087779fa4fad903fa7ee72d2d8bb3605379de8e8112a WatchSource:0}: Error finding container 10db60deed7a7205ca87087779fa4fad903fa7ee72d2d8bb3605379de8e8112a: Status 404 returned error can't find the container with id 10db60deed7a7205ca87087779fa4fad903fa7ee72d2d8bb3605379de8e8112a
Sep 10 20:13:30.421155 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:30.421117    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c" event={"ID":"54c01746-d707-49ff-829f-0cfc1cdc03bb","Type":"ContainerStarted","Data":"5088c81ab2bb419713d15a1aa573e79c8e7ffa9f13e0b6dc018dc80a9429585e"}
Sep 10 20:13:30.421574 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:30.421168    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c" event={"ID":"54c01746-d707-49ff-829f-0cfc1cdc03bb","Type":"ContainerStarted","Data":"10db60deed7a7205ca87087779fa4fad903fa7ee72d2d8bb3605379de8e8112a"}
Sep 10 20:13:30.439280 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:30.439203    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c" podStartSLOduration=1.439187606 podStartE2EDuration="1.439187606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:13:30.43820739 +0000 UTC m=+1151.488735379" watchObservedRunningTime="2026-09-10 20:13:30.439187606 +0000 UTC m=+1151.489715597"
Sep 10 20:13:32.618817 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:32.618765    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:33.356900 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:33.356865    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7748-e6825f"]
Sep 10 20:13:33.360502 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:33.360484    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-e6825f"
Sep 10 20:13:33.363028 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:33.363009    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zcp8m\""
Sep 10 20:13:33.374552 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:33.372499    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7748-e6825f"]
Sep 10 20:13:33.447011 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:33.446969    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4rnq\" (UniqueName: \"kubernetes.io/projected/35178ea2-7e1f-44d3-ae41-e74b8d8a225a-kube-api-access-w4rnq\") pod \"test-curl-7748-e6825f\" (UID: \"35178ea2-7e1f-44d3-ae41-e74b8d8a225a\") " pod="opendatahub/test-curl-7748-e6825f"
Sep 10 20:13:33.548339 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:33.548301    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4rnq\" (UniqueName: \"kubernetes.io/projected/35178ea2-7e1f-44d3-ae41-e74b8d8a225a-kube-api-access-w4rnq\") pod \"test-curl-7748-e6825f\" (UID: \"35178ea2-7e1f-44d3-ae41-e74b8d8a225a\") " pod="opendatahub/test-curl-7748-e6825f"
Sep 10 20:13:33.562204 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:33.562179    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4rnq\" (UniqueName: \"kubernetes.io/projected/35178ea2-7e1f-44d3-ae41-e74b8d8a225a-kube-api-access-w4rnq\") pod \"test-curl-7748-e6825f\" (UID: \"35178ea2-7e1f-44d3-ae41-e74b8d8a225a\") " pod="opendatahub/test-curl-7748-e6825f"
Sep 10 20:13:33.671088 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:33.670993    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-e6825f"
Sep 10 20:13:33.941584 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:33.941558    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7748-e6825f"]
Sep 10 20:13:33.943181 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:13:33.943152    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35178ea2_7e1f_44d3_ae41_e74b8d8a225a.slice/crio-a8dbad1f5cc73293d901540d21884a19273438cd7e20ef5b3205f447e841e706 WatchSource:0}: Error finding container a8dbad1f5cc73293d901540d21884a19273438cd7e20ef5b3205f447e841e706: Status 404 returned error can't find the container with id a8dbad1f5cc73293d901540d21884a19273438cd7e20ef5b3205f447e841e706
Sep 10 20:13:34.444038 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:34.443997    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7748-e6825f" event={"ID":"35178ea2-7e1f-44d3-ae41-e74b8d8a225a","Type":"ContainerStarted","Data":"34a5720f05693866bc3e286724d7d57b1e8d23b5efeb4b6141ab6e64d8c16f3e"}
Sep 10 20:13:34.444038 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:34.444044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7748-e6825f" event={"ID":"35178ea2-7e1f-44d3-ae41-e74b8d8a225a","Type":"ContainerStarted","Data":"a8dbad1f5cc73293d901540d21884a19273438cd7e20ef5b3205f447e841e706"}
Sep 10 20:13:34.461135 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:34.461082    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7748-e6825f" podStartSLOduration=1.461065614 podStartE2EDuration="1.461065614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:13:34.459799847 +0000 UTC m=+1155.510327836" watchObservedRunningTime="2026-09-10 20:13:34.461065614 +0000 UTC m=+1155.511593602"
Sep 10 20:13:35.076851 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:35.076814    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7748-e6825f"]
Sep 10 20:13:35.084098 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:35.084065    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7748-e6825f"]
Sep 10 20:13:35.217033 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:35.216935    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7748-f71934"]
Sep 10 20:13:35.217417 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:35.217398    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35178ea2-7e1f-44d3-ae41-e74b8d8a225a" containerName="test-curl-7748-e6825f"
Sep 10 20:13:35.217505 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:35.217420    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="35178ea2-7e1f-44d3-ae41-e74b8d8a225a" containerName="test-curl-7748-e6825f"
Sep 10 20:13:35.217591 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:35.217515    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="35178ea2-7e1f-44d3-ae41-e74b8d8a225a" containerName="test-curl-7748-e6825f"
Sep 10 20:13:35.220439 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:35.220420    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-f71934"
Sep 10 20:13:35.234265 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:35.234240    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7748-f71934"]
Sep 10 20:13:35.263787 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:35.263755    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2g6f\" (UniqueName: \"kubernetes.io/projected/822787e1-0f32-4347-a153-1ae4e1b2836f-kube-api-access-c2g6f\") pod \"test-curl-7748-f71934\" (UID: \"822787e1-0f32-4347-a153-1ae4e1b2836f\") " pod="opendatahub/test-curl-7748-f71934"
Sep 10 20:13:35.365085 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:35.365047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2g6f\" (UniqueName: \"kubernetes.io/projected/822787e1-0f32-4347-a153-1ae4e1b2836f-kube-api-access-c2g6f\") pod \"test-curl-7748-f71934\" (UID: \"822787e1-0f32-4347-a153-1ae4e1b2836f\") " pod="opendatahub/test-curl-7748-f71934"
Sep 10 20:13:35.374976 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:35.374941    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2g6f\" (UniqueName: \"kubernetes.io/projected/822787e1-0f32-4347-a153-1ae4e1b2836f-kube-api-access-c2g6f\") pod \"test-curl-7748-f71934\" (UID: \"822787e1-0f32-4347-a153-1ae4e1b2836f\") " pod="opendatahub/test-curl-7748-f71934"
Sep 10 20:13:35.449173 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:13:35.449141    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7748-e6825f" container="test-curl-7748-e6825f"
Sep 10 20:13:35.529938 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:35.529908    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-f71934"
Sep 10 20:13:35.804679 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:35.802881    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7748-f71934"]
Sep 10 20:13:35.805769 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:13:35.805740    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod822787e1_0f32_4347_a153_1ae4e1b2836f.slice/crio-5809442ba1fde27b851a538580f81a0e159fe47598942ba14afd34ce36a912c8 WatchSource:0}: Error finding container 5809442ba1fde27b851a538580f81a0e159fe47598942ba14afd34ce36a912c8: Status 404 returned error can't find the container with id 5809442ba1fde27b851a538580f81a0e159fe47598942ba14afd34ce36a912c8
Sep 10 20:13:36.455727 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:36.455690    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7748-f71934" event={"ID":"822787e1-0f32-4347-a153-1ae4e1b2836f","Type":"ContainerStarted","Data":"34edd9b2ede1dcd7f57d45f3a9d1d019c7041c95f81b5900a7f844fa79b7d4d8"}
Sep 10 20:13:36.456204 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:36.455735    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7748-f71934" event={"ID":"822787e1-0f32-4347-a153-1ae4e1b2836f","Type":"ContainerStarted","Data":"5809442ba1fde27b851a538580f81a0e159fe47598942ba14afd34ce36a912c8"}
Sep 10 20:13:36.456204 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:36.455762    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7748-e6825f" podUID="35178ea2-7e1f-44d3-ae41-e74b8d8a225a" containerName="test-curl-7748-e6825f" containerID="cri-o://34a5720f05693866bc3e286724d7d57b1e8d23b5efeb4b6141ab6e64d8c16f3e" gracePeriod=2
Sep 10 20:13:36.473656 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:36.473602    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7748-f71934" podStartSLOduration=1.473586517 podStartE2EDuration="1.473586517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:13:36.471415291 +0000 UTC m=+1157.521943281" watchObservedRunningTime="2026-09-10 20:13:36.473586517 +0000 UTC m=+1157.524114505"
Sep 10 20:13:37.083995 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:37.083957    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7748-f71934"]
Sep 10 20:13:37.087979 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:37.087950    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7748-f71934"]
Sep 10 20:13:37.201185 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:37.201096    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7748-214fc3"]
Sep 10 20:13:37.201508 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:37.201495    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822787e1-0f32-4347-a153-1ae4e1b2836f" containerName="test-curl-7748-f71934"
Sep 10 20:13:37.201575 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:37.201510    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="822787e1-0f32-4347-a153-1ae4e1b2836f" containerName="test-curl-7748-f71934"
Sep 10 20:13:37.201615 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:37.201595    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="822787e1-0f32-4347-a153-1ae4e1b2836f" containerName="test-curl-7748-f71934"
Sep 10 20:13:37.205028 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:37.205014    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-214fc3"
Sep 10 20:13:37.208646 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:37.208620    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7748-214fc3"]
Sep 10 20:13:37.286211 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:37.286168    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ch4c\" (UniqueName: \"kubernetes.io/projected/001fef05-fa96-478e-902a-b42dc5cb31d2-kube-api-access-9ch4c\") pod \"test-curl-7748-214fc3\" (UID: \"001fef05-fa96-478e-902a-b42dc5cb31d2\") " pod="opendatahub/test-curl-7748-214fc3"
Sep 10 20:13:37.387375 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:37.387331    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ch4c\" (UniqueName: \"kubernetes.io/projected/001fef05-fa96-478e-902a-b42dc5cb31d2-kube-api-access-9ch4c\") pod \"test-curl-7748-214fc3\" (UID: \"001fef05-fa96-478e-902a-b42dc5cb31d2\") " pod="opendatahub/test-curl-7748-214fc3"
Sep 10 20:13:37.397108 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:37.397073    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ch4c\" (UniqueName: \"kubernetes.io/projected/001fef05-fa96-478e-902a-b42dc5cb31d2-kube-api-access-9ch4c\") pod \"test-curl-7748-214fc3\" (UID: \"001fef05-fa96-478e-902a-b42dc5cb31d2\") " pod="opendatahub/test-curl-7748-214fc3"
Sep 10 20:13:37.460796 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:13:37.460704    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7748-f71934" container="test-curl-7748-f71934"
Sep 10 20:13:37.515780 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:37.515747    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-214fc3"
Sep 10 20:13:37.787644 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:37.787621    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7748-214fc3"]
Sep 10 20:13:37.789521 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:13:37.789489    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod001fef05_fa96_478e_902a_b42dc5cb31d2.slice/crio-24bcdc5611ca661cba6562f389f3f051077ed1cfc2daac7cc49a0d870590c6f9 WatchSource:0}: Error finding container 24bcdc5611ca661cba6562f389f3f051077ed1cfc2daac7cc49a0d870590c6f9: Status 404 returned error can't find the container with id 24bcdc5611ca661cba6562f389f3f051077ed1cfc2daac7cc49a0d870590c6f9
Sep 10 20:13:38.467762 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:38.467732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7748-214fc3" event={"ID":"001fef05-fa96-478e-902a-b42dc5cb31d2","Type":"ContainerStarted","Data":"f02a241332f0580aaa84f486955c671d64e34423eb9ded397da535f58f7a04b0"}
Sep 10 20:13:38.468114 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:38.467776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7748-214fc3" event={"ID":"001fef05-fa96-478e-902a-b42dc5cb31d2","Type":"ContainerStarted","Data":"24bcdc5611ca661cba6562f389f3f051077ed1cfc2daac7cc49a0d870590c6f9"}
Sep 10 20:13:38.468114 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:38.467886    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7748-f71934" podUID="822787e1-0f32-4347-a153-1ae4e1b2836f" containerName="test-curl-7748-f71934" containerID="cri-o://34edd9b2ede1dcd7f57d45f3a9d1d019c7041c95f81b5900a7f844fa79b7d4d8" gracePeriod=2
Sep 10 20:13:38.485177 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:38.485130    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7748-214fc3" podStartSLOduration=1.4851127100000001 podStartE2EDuration="1.48511271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:13:38.484496958 +0000 UTC m=+1159.535024950" watchObservedRunningTime="2026-09-10 20:13:38.48511271 +0000 UTC m=+1159.535640699"
Sep 10 20:13:38.646438 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:38.646412    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-e6825f"
Sep 10 20:13:38.700242 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:38.700207    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4rnq\" (UniqueName: \"kubernetes.io/projected/35178ea2-7e1f-44d3-ae41-e74b8d8a225a-kube-api-access-w4rnq\") pod \"35178ea2-7e1f-44d3-ae41-e74b8d8a225a\" (UID: \"35178ea2-7e1f-44d3-ae41-e74b8d8a225a\") "
Sep 10 20:13:38.702227 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:38.702203    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35178ea2-7e1f-44d3-ae41-e74b8d8a225a-kube-api-access-w4rnq" (OuterVolumeSpecName: "kube-api-access-w4rnq") pod "35178ea2-7e1f-44d3-ae41-e74b8d8a225a" (UID: "35178ea2-7e1f-44d3-ae41-e74b8d8a225a"). InnerVolumeSpecName "kube-api-access-w4rnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:13:38.801379 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:38.801347    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4rnq\" (UniqueName: \"kubernetes.io/projected/35178ea2-7e1f-44d3-ae41-e74b8d8a225a-kube-api-access-w4rnq\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:13:39.067628 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.067511    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7748-214fc3"]
Sep 10 20:13:39.070261 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.070230    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7748-214fc3"]
Sep 10 20:13:39.185538 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.185501    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7748-2895b1"]
Sep 10 20:13:39.185930 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.185917    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001fef05-fa96-478e-902a-b42dc5cb31d2" containerName="test-curl-7748-214fc3"
Sep 10 20:13:39.185975 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.185933    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="001fef05-fa96-478e-902a-b42dc5cb31d2" containerName="test-curl-7748-214fc3"
Sep 10 20:13:39.186014 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.185997    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="001fef05-fa96-478e-902a-b42dc5cb31d2" containerName="test-curl-7748-214fc3"
Sep 10 20:13:39.188906 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.188891    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-2895b1"
Sep 10 20:13:39.198233 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.198203    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7748-2895b1"]
Sep 10 20:13:39.304677 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.304645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh7wf\" (UniqueName: \"kubernetes.io/projected/7d1acf37-ae39-431b-b4f2-ca51764b1fc2-kube-api-access-hh7wf\") pod \"test-curl-7748-2895b1\" (UID: \"7d1acf37-ae39-431b-b4f2-ca51764b1fc2\") " pod="opendatahub/test-curl-7748-2895b1"
Sep 10 20:13:39.404299 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.404216    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35178ea2-7e1f-44d3-ae41-e74b8d8a225a" path="/var/lib/kubelet/pods/35178ea2-7e1f-44d3-ae41-e74b8d8a225a/volumes"
Sep 10 20:13:39.405265 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.405241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh7wf\" (UniqueName: \"kubernetes.io/projected/7d1acf37-ae39-431b-b4f2-ca51764b1fc2-kube-api-access-hh7wf\") pod \"test-curl-7748-2895b1\" (UID: \"7d1acf37-ae39-431b-b4f2-ca51764b1fc2\") " pod="opendatahub/test-curl-7748-2895b1"
Sep 10 20:13:39.417441 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.417417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh7wf\" (UniqueName: \"kubernetes.io/projected/7d1acf37-ae39-431b-b4f2-ca51764b1fc2-kube-api-access-hh7wf\") pod \"test-curl-7748-2895b1\" (UID: \"7d1acf37-ae39-431b-b4f2-ca51764b1fc2\") " pod="opendatahub/test-curl-7748-2895b1"
Sep 10 20:13:39.474686 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.474651    2561 generic.go:356] "Generic (PLEG): container finished" podID="35178ea2-7e1f-44d3-ae41-e74b8d8a225a" containerID="34a5720f05693866bc3e286724d7d57b1e8d23b5efeb4b6141ab6e64d8c16f3e" exitCode=137
Sep 10 20:13:39.475133 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.474700    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-e6825f"
Sep 10 20:13:39.475133 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.474759    2561 scope.go:117] "RemoveContainer" containerID="34a5720f05693866bc3e286724d7d57b1e8d23b5efeb4b6141ab6e64d8c16f3e"
Sep 10 20:13:39.475133 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:13:39.474903    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7748-214fc3" container="test-curl-7748-214fc3"
Sep 10 20:13:39.484559 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.484520    2561 scope.go:117] "RemoveContainer" containerID="34a5720f05693866bc3e286724d7d57b1e8d23b5efeb4b6141ab6e64d8c16f3e"
Sep 10 20:13:39.484842 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:13:39.484822    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a5720f05693866bc3e286724d7d57b1e8d23b5efeb4b6141ab6e64d8c16f3e\": container with ID starting with 34a5720f05693866bc3e286724d7d57b1e8d23b5efeb4b6141ab6e64d8c16f3e not found: ID does not exist" containerID="34a5720f05693866bc3e286724d7d57b1e8d23b5efeb4b6141ab6e64d8c16f3e"
Sep 10 20:13:39.484898 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.484853    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a5720f05693866bc3e286724d7d57b1e8d23b5efeb4b6141ab6e64d8c16f3e"} err="failed to get container status \"34a5720f05693866bc3e286724d7d57b1e8d23b5efeb4b6141ab6e64d8c16f3e\": rpc error: code = NotFound desc = could not find container \"34a5720f05693866bc3e286724d7d57b1e8d23b5efeb4b6141ab6e64d8c16f3e\": container with ID starting with 34a5720f05693866bc3e286724d7d57b1e8d23b5efeb4b6141ab6e64d8c16f3e not found: ID does not exist"
Sep 10 20:13:39.499031 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.499006    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-2895b1"
Sep 10 20:13:39.559489 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.559457    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:39.559661 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.559509    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:39.577999 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.577965    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:39.777430 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:39.777403    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7748-2895b1"]
Sep 10 20:13:39.779120 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:13:39.779094    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d1acf37_ae39_431b_b4f2_ca51764b1fc2.slice/crio-2206b71bd9854eaae670652254d6885b85dfd791741296eafb06cbd86f98b1f6 WatchSource:0}: Error finding container 2206b71bd9854eaae670652254d6885b85dfd791741296eafb06cbd86f98b1f6: Status 404 returned error can't find the container with id 2206b71bd9854eaae670652254d6885b85dfd791741296eafb06cbd86f98b1f6
Sep 10 20:13:40.482711 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:40.482684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7748-2895b1" event={"ID":"7d1acf37-ae39-431b-b4f2-ca51764b1fc2","Type":"ContainerStarted","Data":"703087ad78643881b6a615b6fb002a57ace8ebbdc8f86fc3d1fea32d0c456605"}
Sep 10 20:13:40.483102 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:40.482722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7748-2895b1" event={"ID":"7d1acf37-ae39-431b-b4f2-ca51764b1fc2","Type":"ContainerStarted","Data":"2206b71bd9854eaae670652254d6885b85dfd791741296eafb06cbd86f98b1f6"}
Sep 10 20:13:40.483102 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:40.483023    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7748-214fc3" podUID="001fef05-fa96-478e-902a-b42dc5cb31d2" containerName="test-curl-7748-214fc3" containerID="cri-o://f02a241332f0580aaa84f486955c671d64e34423eb9ded397da535f58f7a04b0" gracePeriod=2
Sep 10 20:13:40.494890 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:40.494868    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:13:40.500125 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:40.500083    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7748-2895b1" podStartSLOduration=1.500067651 podStartE2EDuration="1.500067651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:13:40.498348997 +0000 UTC m=+1161.548876987" watchObservedRunningTime="2026-09-10 20:13:40.500067651 +0000 UTC m=+1161.550595640"
Sep 10 20:13:40.663268 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:40.663240    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-f71934"
Sep 10 20:13:40.716797 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:40.716768    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2g6f\" (UniqueName: \"kubernetes.io/projected/822787e1-0f32-4347-a153-1ae4e1b2836f-kube-api-access-c2g6f\") pod \"822787e1-0f32-4347-a153-1ae4e1b2836f\" (UID: \"822787e1-0f32-4347-a153-1ae4e1b2836f\") "
Sep 10 20:13:40.719065 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:40.719041    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822787e1-0f32-4347-a153-1ae4e1b2836f-kube-api-access-c2g6f" (OuterVolumeSpecName: "kube-api-access-c2g6f") pod "822787e1-0f32-4347-a153-1ae4e1b2836f" (UID: "822787e1-0f32-4347-a153-1ae4e1b2836f"). InnerVolumeSpecName "kube-api-access-c2g6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:13:40.818365 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:40.818334    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2g6f\" (UniqueName: \"kubernetes.io/projected/822787e1-0f32-4347-a153-1ae4e1b2836f-kube-api-access-c2g6f\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:13:41.134412 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:41.134378    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7748-2895b1"]
Sep 10 20:13:41.139076 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:41.139049    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7748-2895b1"]
Sep 10 20:13:41.404833 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:41.404757    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="822787e1-0f32-4347-a153-1ae4e1b2836f" path="/var/lib/kubelet/pods/822787e1-0f32-4347-a153-1ae4e1b2836f/volumes"
Sep 10 20:13:41.489751 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:41.489713    2561 generic.go:356] "Generic (PLEG): container finished" podID="822787e1-0f32-4347-a153-1ae4e1b2836f" containerID="34edd9b2ede1dcd7f57d45f3a9d1d019c7041c95f81b5900a7f844fa79b7d4d8" exitCode=137
Sep 10 20:13:41.490203 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:41.489774    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-f71934"
Sep 10 20:13:41.490203 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:41.489810    2561 scope.go:117] "RemoveContainer" containerID="34edd9b2ede1dcd7f57d45f3a9d1d019c7041c95f81b5900a7f844fa79b7d4d8"
Sep 10 20:13:41.490203 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:13:41.490135    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7748-2895b1" container="test-curl-7748-2895b1"
Sep 10 20:13:41.498932 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:41.498915    2561 scope.go:117] "RemoveContainer" containerID="34edd9b2ede1dcd7f57d45f3a9d1d019c7041c95f81b5900a7f844fa79b7d4d8"
Sep 10 20:13:41.499187 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:13:41.499168    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34edd9b2ede1dcd7f57d45f3a9d1d019c7041c95f81b5900a7f844fa79b7d4d8\": container with ID starting with 34edd9b2ede1dcd7f57d45f3a9d1d019c7041c95f81b5900a7f844fa79b7d4d8 not found: ID does not exist" containerID="34edd9b2ede1dcd7f57d45f3a9d1d019c7041c95f81b5900a7f844fa79b7d4d8"
Sep 10 20:13:41.499253 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:41.499199    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34edd9b2ede1dcd7f57d45f3a9d1d019c7041c95f81b5900a7f844fa79b7d4d8"} err="failed to get container status \"34edd9b2ede1dcd7f57d45f3a9d1d019c7041c95f81b5900a7f844fa79b7d4d8\": rpc error: code = NotFound desc = could not find container \"34edd9b2ede1dcd7f57d45f3a9d1d019c7041c95f81b5900a7f844fa79b7d4d8\": container with ID starting with 34edd9b2ede1dcd7f57d45f3a9d1d019c7041c95f81b5900a7f844fa79b7d4d8 not found: ID does not exist"
Sep 10 20:13:41.747807 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:41.747716    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:42.057899 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:42.057870    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:42.496697 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:42.496671    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7748-2895b1" podUID="7d1acf37-ae39-431b-b4f2-ca51764b1fc2" containerName="test-curl-7748-2895b1" containerID="cri-o://703087ad78643881b6a615b6fb002a57ace8ebbdc8f86fc3d1fea32d0c456605" gracePeriod=2
Sep 10 20:13:42.654699 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:42.654671    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-214fc3"
Sep 10 20:13:42.735119 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:42.735084    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ch4c\" (UniqueName: \"kubernetes.io/projected/001fef05-fa96-478e-902a-b42dc5cb31d2-kube-api-access-9ch4c\") pod \"001fef05-fa96-478e-902a-b42dc5cb31d2\" (UID: \"001fef05-fa96-478e-902a-b42dc5cb31d2\") "
Sep 10 20:13:42.737339 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:42.737283    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001fef05-fa96-478e-902a-b42dc5cb31d2-kube-api-access-9ch4c" (OuterVolumeSpecName: "kube-api-access-9ch4c") pod "001fef05-fa96-478e-902a-b42dc5cb31d2" (UID: "001fef05-fa96-478e-902a-b42dc5cb31d2"). InnerVolumeSpecName "kube-api-access-9ch4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:13:42.812761 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:42.812727    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:42.836726 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:42.836691    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ch4c\" (UniqueName: \"kubernetes.io/projected/001fef05-fa96-478e-902a-b42dc5cb31d2-kube-api-access-9ch4c\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:13:43.404289 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:43.404259    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="001fef05-fa96-478e-902a-b42dc5cb31d2" path="/var/lib/kubelet/pods/001fef05-fa96-478e-902a-b42dc5cb31d2/volumes"
Sep 10 20:13:43.506464 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:43.506431    2561 generic.go:356] "Generic (PLEG): container finished" podID="001fef05-fa96-478e-902a-b42dc5cb31d2" containerID="f02a241332f0580aaa84f486955c671d64e34423eb9ded397da535f58f7a04b0" exitCode=137
Sep 10 20:13:43.506936 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:43.506477    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-214fc3"
Sep 10 20:13:43.506936 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:43.506515    2561 scope.go:117] "RemoveContainer" containerID="f02a241332f0580aaa84f486955c671d64e34423eb9ded397da535f58f7a04b0"
Sep 10 20:13:43.515235 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:43.515217    2561 scope.go:117] "RemoveContainer" containerID="f02a241332f0580aaa84f486955c671d64e34423eb9ded397da535f58f7a04b0"
Sep 10 20:13:43.515504 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:13:43.515489    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f02a241332f0580aaa84f486955c671d64e34423eb9ded397da535f58f7a04b0\": container with ID starting with f02a241332f0580aaa84f486955c671d64e34423eb9ded397da535f58f7a04b0 not found: ID does not exist" containerID="f02a241332f0580aaa84f486955c671d64e34423eb9ded397da535f58f7a04b0"
Sep 10 20:13:43.515588 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:43.515510    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02a241332f0580aaa84f486955c671d64e34423eb9ded397da535f58f7a04b0"} err="failed to get container status \"f02a241332f0580aaa84f486955c671d64e34423eb9ded397da535f58f7a04b0\": rpc error: code = NotFound desc = could not find container \"f02a241332f0580aaa84f486955c671d64e34423eb9ded397da535f58f7a04b0\": container with ID starting with f02a241332f0580aaa84f486955c671d64e34423eb9ded397da535f58f7a04b0 not found: ID does not exist"
Sep 10 20:13:44.665742 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:44.665715    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-2895b1"
Sep 10 20:13:44.754804 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:44.754707    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh7wf\" (UniqueName: \"kubernetes.io/projected/7d1acf37-ae39-431b-b4f2-ca51764b1fc2-kube-api-access-hh7wf\") pod \"7d1acf37-ae39-431b-b4f2-ca51764b1fc2\" (UID: \"7d1acf37-ae39-431b-b4f2-ca51764b1fc2\") "
Sep 10 20:13:44.756872 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:44.756852    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1acf37-ae39-431b-b4f2-ca51764b1fc2-kube-api-access-hh7wf" (OuterVolumeSpecName: "kube-api-access-hh7wf") pod "7d1acf37-ae39-431b-b4f2-ca51764b1fc2" (UID: "7d1acf37-ae39-431b-b4f2-ca51764b1fc2"). InnerVolumeSpecName "kube-api-access-hh7wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:13:44.855376 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:44.855336    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh7wf\" (UniqueName: \"kubernetes.io/projected/7d1acf37-ae39-431b-b4f2-ca51764b1fc2-kube-api-access-hh7wf\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:13:44.879042 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:44.879010    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:45.013066 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:45.012977    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:45.403505 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:45.403474    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d1acf37-ae39-431b-b4f2-ca51764b1fc2" path="/var/lib/kubelet/pods/7d1acf37-ae39-431b-b4f2-ca51764b1fc2/volumes"
Sep 10 20:13:45.520518 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:45.520483    2561 generic.go:356] "Generic (PLEG): container finished" podID="7d1acf37-ae39-431b-b4f2-ca51764b1fc2" containerID="703087ad78643881b6a615b6fb002a57ace8ebbdc8f86fc3d1fea32d0c456605" exitCode=137
Sep 10 20:13:45.520698 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:45.520542    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-2895b1"
Sep 10 20:13:45.520698 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:45.520572    2561 scope.go:117] "RemoveContainer" containerID="703087ad78643881b6a615b6fb002a57ace8ebbdc8f86fc3d1fea32d0c456605"
Sep 10 20:13:45.529311 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:45.529295    2561 scope.go:117] "RemoveContainer" containerID="703087ad78643881b6a615b6fb002a57ace8ebbdc8f86fc3d1fea32d0c456605"
Sep 10 20:13:45.529579 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:13:45.529560    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703087ad78643881b6a615b6fb002a57ace8ebbdc8f86fc3d1fea32d0c456605\": container with ID starting with 703087ad78643881b6a615b6fb002a57ace8ebbdc8f86fc3d1fea32d0c456605 not found: ID does not exist" containerID="703087ad78643881b6a615b6fb002a57ace8ebbdc8f86fc3d1fea32d0c456605"
Sep 10 20:13:45.529653 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:45.529592    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703087ad78643881b6a615b6fb002a57ace8ebbdc8f86fc3d1fea32d0c456605"} err="failed to get container status \"703087ad78643881b6a615b6fb002a57ace8ebbdc8f86fc3d1fea32d0c456605\": rpc error: code = NotFound desc = could not find container \"703087ad78643881b6a615b6fb002a57ace8ebbdc8f86fc3d1fea32d0c456605\": container with ID starting with 703087ad78643881b6a615b6fb002a57ace8ebbdc8f86fc3d1fea32d0c456605 not found: ID does not exist"
Sep 10 20:13:46.047789 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.047757    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"]
Sep 10 20:13:46.048153 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.048142    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1acf37-ae39-431b-b4f2-ca51764b1fc2" containerName="test-curl-7748-2895b1"
Sep 10 20:13:46.048153 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.048153    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1acf37-ae39-431b-b4f2-ca51764b1fc2" containerName="test-curl-7748-2895b1"
Sep 10 20:13:46.048238 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.048210    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d1acf37-ae39-431b-b4f2-ca51764b1fc2" containerName="test-curl-7748-2895b1"
Sep 10 20:13:46.052750 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.052728    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:46.055636 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.055606    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e88b3e4f-serving-cert\""
Sep 10 20:13:46.074781 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.074750    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"]
Sep 10 20:13:46.099884 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.099857    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:46.168292 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.168253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-openshift-service-ca\") pod \"maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:46.168470 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.168324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-metrics-certs\") pod \"maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:46.168470 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.168367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl88m\" (UniqueName: \"kubernetes.io/projected/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-kube-api-access-nl88m\") pod \"maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:46.168470 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.168396    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-maas-api-tls\") pod \"maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:46.202767 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.202736    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:46.269382 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.269343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-metrics-certs\") pod \"maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:46.269593 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.269386    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl88m\" (UniqueName: \"kubernetes.io/projected/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-kube-api-access-nl88m\") pod \"maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:46.269593 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.269428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-maas-api-tls\") pod \"maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:46.269593 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.269522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-openshift-service-ca\") pod \"maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:46.270160 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.270132    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-openshift-service-ca\") pod \"maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:46.272018 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.272000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-metrics-certs\") pod \"maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:46.272102 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.272040    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-maas-api-tls\") pod \"maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:46.285243 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.285214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl88m\" (UniqueName: \"kubernetes.io/projected/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-kube-api-access-nl88m\") pod \"maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:46.323353 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.323275    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:46.363467 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.363433    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:46.661434 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.661342    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"]
Sep 10 20:13:46.664320 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:13:46.664294    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a02b81c_f1f6_41e9_986e_ce0f7ecba1a2.slice/crio-8384c712eb8f1632ce6834bb83212cab382079e096cae0c9a5b6412a0c0b258e WatchSource:0}: Error finding container 8384c712eb8f1632ce6834bb83212cab382079e096cae0c9a5b6412a0c0b258e: Status 404 returned error can't find the container with id 8384c712eb8f1632ce6834bb83212cab382079e096cae0c9a5b6412a0c0b258e
Sep 10 20:13:46.923966 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:46.923885    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:47.014112 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:47.014073    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:47.536827 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:47.536790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7" event={"ID":"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2","Type":"ContainerStarted","Data":"63221caa239c3db1c1a0af1fce06e8726b9fb719858fb022af9437c8b3f1d79c"}
Sep 10 20:13:47.537218 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:47.536849    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7" event={"ID":"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2","Type":"ContainerStarted","Data":"8384c712eb8f1632ce6834bb83212cab382079e096cae0c9a5b6412a0c0b258e"}
Sep 10 20:13:47.537218 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:47.536983    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:13:47.564996 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:47.564938    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7" podStartSLOduration=1.058535956 podStartE2EDuration="1.564916493s" totalImagesPullingTime="506.380537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:13:47.564457577 +0000 UTC m=+1168.614985569" watchObservedRunningTime="2026-09-10 20:13:47.564916493 +0000 UTC m=+1168.615444481"
Sep 10 20:13:48.792305 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:48.792265    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:13:53.548607 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:13:53.548579    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:14:07.379516 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:07.379467    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:14:09.848052 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:09.848018    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:14:20.847021 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:20.846916    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 20:14:20.854491 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:20.852386    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 20:14:24.254603 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.254572    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"]
Sep 10 20:14:24.257922 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.257904    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.261385 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.261353    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e88b3e4f\"/\"kube-root-ca.crt\""
Sep 10 20:14:24.261499 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.261449    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e88b3e4f\"/\"openshift-service-ca.crt\""
Sep 10 20:14:24.262181 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.262164    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e88b3e4f\"/\"default-dockercfg-c8rg8\""
Sep 10 20:14:24.262288 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.262220    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e88b3e4f\"/\"e2e-lifecycle-model-e88b3e4f-kserve-self-signed-certs\""
Sep 10 20:14:24.284276 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.284246    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"]
Sep 10 20:14:24.305985 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.305948    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:14:24.335585 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.335543    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-model-cache\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.335780 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.335591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhh7v\" (UniqueName: \"kubernetes.io/projected/1c753567-2156-43dc-bc8d-5cae6bf81b55-kube-api-access-fhh7v\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.335780 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.335642    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-home\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.335780 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.335663    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-dshm\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.335780 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.335694    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-tmp-dir\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.335937 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.335783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c753567-2156-43dc-bc8d-5cae6bf81b55-tls-certs\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.436721 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.436659    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-model-cache\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.436721 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.436726    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhh7v\" (UniqueName: \"kubernetes.io/projected/1c753567-2156-43dc-bc8d-5cae6bf81b55-kube-api-access-fhh7v\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.436990 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.436777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-home\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.436990 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.436802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-dshm\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.436990 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.436846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-tmp-dir\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.436990 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.436931    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c753567-2156-43dc-bc8d-5cae6bf81b55-tls-certs\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.437228 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.437152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-home\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.437228 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.437191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-model-cache\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.437310 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.437298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-tmp-dir\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.439260 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.439227    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-dshm\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.439476 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.439460    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c753567-2156-43dc-bc8d-5cae6bf81b55-tls-certs\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.446388 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.446364    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhh7v\" (UniqueName: \"kubernetes.io/projected/1c753567-2156-43dc-bc8d-5cae6bf81b55-kube-api-access-fhh7v\") pod \"e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") " pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.567118 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.567074    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:24.843136 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:24.843109    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"]
Sep 10 20:14:24.844686 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:14:24.844658    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c753567_2156_43dc_bc8d_5cae6bf81b55.slice/crio-ff072bddc1a831d6e502e6b039941ca4caa115ed6f04f0960d2300002ca1ffca WatchSource:0}: Error finding container ff072bddc1a831d6e502e6b039941ca4caa115ed6f04f0960d2300002ca1ffca: Status 404 returned error can't find the container with id ff072bddc1a831d6e502e6b039941ca4caa115ed6f04f0960d2300002ca1ffca
Sep 10 20:14:25.754091 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:25.754057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq" event={"ID":"1c753567-2156-43dc-bc8d-5cae6bf81b55","Type":"ContainerStarted","Data":"12b07c6abc8ef1dc35433c1bb69e66a8c4da583ecd86ab72c4e9433276c217f7"}
Sep 10 20:14:25.754506 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:25.754103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq" event={"ID":"1c753567-2156-43dc-bc8d-5cae6bf81b55","Type":"ContainerStarted","Data":"ff072bddc1a831d6e502e6b039941ca4caa115ed6f04f0960d2300002ca1ffca"}
Sep 10 20:14:25.776866 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:25.776818    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq" podStartSLOduration=1.7768028359999999 podStartE2EDuration="1.776802836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:14:25.7743535 +0000 UTC m=+1206.824881490" watchObservedRunningTime="2026-09-10 20:14:25.776802836 +0000 UTC m=+1206.827330841"
Sep 10 20:14:34.567210 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:34.567169    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:34.567210 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:34.567224    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:34.580004 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:34.579979    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:14:34.814987 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:14:34.814960    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:15:14.995787 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:14.995700    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"]
Sep 10 20:15:14.996390 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:14.996006    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7" podUID="3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2" containerName="maas-api" containerID="cri-o://63221caa239c3db1c1a0af1fce06e8726b9fb719858fb022af9437c8b3f1d79c" gracePeriod=30
Sep 10 20:15:15.034270 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:15.034233    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:15.269300 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:15.269274    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:15:15.451718 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:15.451684    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-metrics-certs\") pod \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") "
Sep 10 20:15:15.451901 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:15.451758    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl88m\" (UniqueName: \"kubernetes.io/projected/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-kube-api-access-nl88m\") pod \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") "
Sep 10 20:15:15.451901 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:15.451789    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-openshift-service-ca\") pod \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") "
Sep 10 20:15:15.451901 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:15.451847    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-maas-api-tls\") pod \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\" (UID: \"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2\") "
Sep 10 20:15:15.452268 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:15.452238    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2" (UID: "3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:15:15.454146 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:15.454113    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2" (UID: "3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:15:15.454146 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:15.454134    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-kube-api-access-nl88m" (OuterVolumeSpecName: "kube-api-access-nl88m") pod "3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2" (UID: "3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2"). InnerVolumeSpecName "kube-api-access-nl88m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:15:15.454311 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:15.454232    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2" (UID: "3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:15:15.553314 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:15.553274    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:15.553314 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:15.553305    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:15.553314 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:15.553318    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl88m\" (UniqueName: \"kubernetes.io/projected/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-kube-api-access-nl88m\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:15.553576 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:15.553332    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:16.041676 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:16.041638    2561 generic.go:356] "Generic (PLEG): container finished" podID="3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2" containerID="63221caa239c3db1c1a0af1fce06e8726b9fb719858fb022af9437c8b3f1d79c" exitCode=0
Sep 10 20:15:16.042103 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:16.041687    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"
Sep 10 20:15:16.042103 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:16.041686    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7" event={"ID":"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2","Type":"ContainerDied","Data":"63221caa239c3db1c1a0af1fce06e8726b9fb719858fb022af9437c8b3f1d79c"}
Sep 10 20:15:16.042103 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:16.041742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7" event={"ID":"3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2","Type":"ContainerDied","Data":"8384c712eb8f1632ce6834bb83212cab382079e096cae0c9a5b6412a0c0b258e"}
Sep 10 20:15:16.042103 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:16.041785    2561 scope.go:117] "RemoveContainer" containerID="63221caa239c3db1c1a0af1fce06e8726b9fb719858fb022af9437c8b3f1d79c"
Sep 10 20:15:16.052991 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:16.052972    2561 scope.go:117] "RemoveContainer" containerID="63221caa239c3db1c1a0af1fce06e8726b9fb719858fb022af9437c8b3f1d79c"
Sep 10 20:15:16.053399 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:15:16.053368    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63221caa239c3db1c1a0af1fce06e8726b9fb719858fb022af9437c8b3f1d79c\": container with ID starting with 63221caa239c3db1c1a0af1fce06e8726b9fb719858fb022af9437c8b3f1d79c not found: ID does not exist" containerID="63221caa239c3db1c1a0af1fce06e8726b9fb719858fb022af9437c8b3f1d79c"
Sep 10 20:15:16.053476 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:16.053409    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63221caa239c3db1c1a0af1fce06e8726b9fb719858fb022af9437c8b3f1d79c"} err="failed to get container status \"63221caa239c3db1c1a0af1fce06e8726b9fb719858fb022af9437c8b3f1d79c\": rpc error: code = NotFound desc = could not find container \"63221caa239c3db1c1a0af1fce06e8726b9fb719858fb022af9437c8b3f1d79c\": container with ID starting with 63221caa239c3db1c1a0af1fce06e8726b9fb719858fb022af9437c8b3f1d79c not found: ID does not exist"
Sep 10 20:15:16.066932 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:16.066896    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"]
Sep 10 20:15:16.069173 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:16.069152    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e88b3e4f-6df9679544-zxpt7"]
Sep 10 20:15:16.246323 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:16.246284    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:17.404470 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:17.404437    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2" path="/var/lib/kubelet/pods/3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2/volumes"
Sep 10 20:15:18.355831 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:18.355797    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:22.656261 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:22.656230    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:23.336156 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:23.336119    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"]
Sep 10 20:15:23.430136 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:23.430105    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:31.888545 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:31.888490    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7751-e66a4c"]
Sep 10 20:15:31.888944 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:31.888880    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2" containerName="maas-api"
Sep 10 20:15:31.888944 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:31.888893    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2" containerName="maas-api"
Sep 10 20:15:31.889018 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:31.888959    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a02b81c-f1f6-41e9-986e-ce0f7ecba1a2" containerName="maas-api"
Sep 10 20:15:31.893306 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:31.893290    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-e66a4c"
Sep 10 20:15:31.897207 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:31.897188    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zcp8m\""
Sep 10 20:15:31.901550 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:31.901455    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7751-e66a4c"]
Sep 10 20:15:31.995202 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:31.995170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99tnn\" (UniqueName: \"kubernetes.io/projected/8b4a09cb-18d5-46e5-8942-e0fbfe816a19-kube-api-access-99tnn\") pod \"test-curl-7751-e66a4c\" (UID: \"8b4a09cb-18d5-46e5-8942-e0fbfe816a19\") " pod="opendatahub/test-curl-7751-e66a4c"
Sep 10 20:15:32.096669 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:32.096634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99tnn\" (UniqueName: \"kubernetes.io/projected/8b4a09cb-18d5-46e5-8942-e0fbfe816a19-kube-api-access-99tnn\") pod \"test-curl-7751-e66a4c\" (UID: \"8b4a09cb-18d5-46e5-8942-e0fbfe816a19\") " pod="opendatahub/test-curl-7751-e66a4c"
Sep 10 20:15:32.106100 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:32.106066    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99tnn\" (UniqueName: \"kubernetes.io/projected/8b4a09cb-18d5-46e5-8942-e0fbfe816a19-kube-api-access-99tnn\") pod \"test-curl-7751-e66a4c\" (UID: \"8b4a09cb-18d5-46e5-8942-e0fbfe816a19\") " pod="opendatahub/test-curl-7751-e66a4c"
Sep 10 20:15:32.204702 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:32.204611    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-e66a4c"
Sep 10 20:15:32.482889 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:32.482857    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7751-e66a4c"]
Sep 10 20:15:32.484347 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:15:32.484317    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b4a09cb_18d5_46e5_8942_e0fbfe816a19.slice/crio-2799f33f3d3279214a197ef9337172ef9c7d5889f9db377d66b6ab399db0e0dd WatchSource:0}: Error finding container 2799f33f3d3279214a197ef9337172ef9c7d5889f9db377d66b6ab399db0e0dd: Status 404 returned error can't find the container with id 2799f33f3d3279214a197ef9337172ef9c7d5889f9db377d66b6ab399db0e0dd
Sep 10 20:15:33.135300 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:33.135260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7751-e66a4c" event={"ID":"8b4a09cb-18d5-46e5-8942-e0fbfe816a19","Type":"ContainerStarted","Data":"47c5cf8df501a15ec4eda120e0559df77c1b4c89f9a54fc512f2102fdb3e4469"}
Sep 10 20:15:33.135300 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:33.135303    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7751-e66a4c" event={"ID":"8b4a09cb-18d5-46e5-8942-e0fbfe816a19","Type":"ContainerStarted","Data":"2799f33f3d3279214a197ef9337172ef9c7d5889f9db377d66b6ab399db0e0dd"}
Sep 10 20:15:33.150270 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:33.150213    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7751-e66a4c" podStartSLOduration=2.150194758 podStartE2EDuration="2.150194758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:15:33.148957412 +0000 UTC m=+1274.199485400" watchObservedRunningTime="2026-09-10 20:15:33.150194758 +0000 UTC m=+1274.200722746"
Sep 10 20:15:33.769009 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:33.768970    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7751-e66a4c"]
Sep 10 20:15:33.773636 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:33.773603    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7751-e66a4c"]
Sep 10 20:15:34.140174 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:15:34.140145    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7751-e66a4c" container="test-curl-7751-e66a4c"
Sep 10 20:15:35.130140 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:35.130104    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"]
Sep 10 20:15:35.145107 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:35.145068    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7751-e66a4c" podUID="8b4a09cb-18d5-46e5-8942-e0fbfe816a19" containerName="test-curl-7751-e66a4c" containerID="cri-o://47c5cf8df501a15ec4eda120e0559df77c1b4c89f9a54fc512f2102fdb3e4469" gracePeriod=2
Sep 10 20:15:35.150284 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:15:35.150249    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7751-e66a4c\" is forbidden: User \"system:node:ip-10-0-140-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-111.ec2.internal' and this object" podUID="8b4a09cb-18d5-46e5-8942-e0fbfe816a19" pod="opendatahub/test-curl-7751-e66a4c"
Sep 10 20:15:35.156487 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:35.156464    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:36.161319 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:36.161278    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"]
Sep 10 20:15:36.187550 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:36.187505    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:37.313764 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:37.313732    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-e66a4c"
Sep 10 20:15:37.447024 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:37.446927    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99tnn\" (UniqueName: \"kubernetes.io/projected/8b4a09cb-18d5-46e5-8942-e0fbfe816a19-kube-api-access-99tnn\") pod \"8b4a09cb-18d5-46e5-8942-e0fbfe816a19\" (UID: \"8b4a09cb-18d5-46e5-8942-e0fbfe816a19\") "
Sep 10 20:15:37.449284 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:37.449249    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4a09cb-18d5-46e5-8942-e0fbfe816a19-kube-api-access-99tnn" (OuterVolumeSpecName: "kube-api-access-99tnn") pod "8b4a09cb-18d5-46e5-8942-e0fbfe816a19" (UID: "8b4a09cb-18d5-46e5-8942-e0fbfe816a19"). InnerVolumeSpecName "kube-api-access-99tnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:15:37.547922 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:37.547890    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99tnn\" (UniqueName: \"kubernetes.io/projected/8b4a09cb-18d5-46e5-8942-e0fbfe816a19-kube-api-access-99tnn\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:38.166336 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.166297    2561 generic.go:356] "Generic (PLEG): container finished" podID="8b4a09cb-18d5-46e5-8942-e0fbfe816a19" containerID="47c5cf8df501a15ec4eda120e0559df77c1b4c89f9a54fc512f2102fdb3e4469" exitCode=137
Sep 10 20:15:38.166552 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.166352    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-e66a4c"
Sep 10 20:15:38.166552 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.166365    2561 scope.go:117] "RemoveContainer" containerID="47c5cf8df501a15ec4eda120e0559df77c1b4c89f9a54fc512f2102fdb3e4469"
Sep 10 20:15:38.176018 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.175996    2561 scope.go:117] "RemoveContainer" containerID="47c5cf8df501a15ec4eda120e0559df77c1b4c89f9a54fc512f2102fdb3e4469"
Sep 10 20:15:38.176336 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:15:38.176318    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c5cf8df501a15ec4eda120e0559df77c1b4c89f9a54fc512f2102fdb3e4469\": container with ID starting with 47c5cf8df501a15ec4eda120e0559df77c1b4c89f9a54fc512f2102fdb3e4469 not found: ID does not exist" containerID="47c5cf8df501a15ec4eda120e0559df77c1b4c89f9a54fc512f2102fdb3e4469"
Sep 10 20:15:38.176416 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.176345    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c5cf8df501a15ec4eda120e0559df77c1b4c89f9a54fc512f2102fdb3e4469"} err="failed to get container status \"47c5cf8df501a15ec4eda120e0559df77c1b4c89f9a54fc512f2102fdb3e4469\": rpc error: code = NotFound desc = could not find container \"47c5cf8df501a15ec4eda120e0559df77c1b4c89f9a54fc512f2102fdb3e4469\": container with ID starting with 47c5cf8df501a15ec4eda120e0559df77c1b4c89f9a54fc512f2102fdb3e4469 not found: ID does not exist"
Sep 10 20:15:38.356819 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.356769    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq" podUID="1c753567-2156-43dc-bc8d-5cae6bf81b55" containerName="main" containerID="cri-o://12b07c6abc8ef1dc35433c1bb69e66a8c4da583ecd86ab72c4e9433276c217f7" gracePeriod=45
Sep 10 20:15:38.617921 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.617896    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:15:38.659215 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.659185    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhh7v\" (UniqueName: \"kubernetes.io/projected/1c753567-2156-43dc-bc8d-5cae6bf81b55-kube-api-access-fhh7v\") pod \"1c753567-2156-43dc-bc8d-5cae6bf81b55\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") "
Sep 10 20:15:38.659390 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.659254    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-model-cache\") pod \"1c753567-2156-43dc-bc8d-5cae6bf81b55\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") "
Sep 10 20:15:38.659390 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.659285    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-dshm\") pod \"1c753567-2156-43dc-bc8d-5cae6bf81b55\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") "
Sep 10 20:15:38.660135 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.659870    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-tmp-dir\") pod \"1c753567-2156-43dc-bc8d-5cae6bf81b55\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") "
Sep 10 20:15:38.660135 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.659892    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c753567-2156-43dc-bc8d-5cae6bf81b55" (UID: "1c753567-2156-43dc-bc8d-5cae6bf81b55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:15:38.660135 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.659934    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c753567-2156-43dc-bc8d-5cae6bf81b55-tls-certs\") pod \"1c753567-2156-43dc-bc8d-5cae6bf81b55\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") "
Sep 10 20:15:38.660135 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.660049    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-home\") pod \"1c753567-2156-43dc-bc8d-5cae6bf81b55\" (UID: \"1c753567-2156-43dc-bc8d-5cae6bf81b55\") "
Sep 10 20:15:38.660135 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.660071    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c753567-2156-43dc-bc8d-5cae6bf81b55" (UID: "1c753567-2156-43dc-bc8d-5cae6bf81b55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:15:38.662583 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.660700    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-home" (OuterVolumeSpecName: "home") pod "1c753567-2156-43dc-bc8d-5cae6bf81b55" (UID: "1c753567-2156-43dc-bc8d-5cae6bf81b55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:15:38.662583 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.661410    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:38.662583 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.661505    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:38.662583 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.661519    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:38.668555 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.663159    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-dshm" (OuterVolumeSpecName: "dshm") pod "1c753567-2156-43dc-bc8d-5cae6bf81b55" (UID: "1c753567-2156-43dc-bc8d-5cae6bf81b55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:15:38.671512 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.671481    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c753567-2156-43dc-bc8d-5cae6bf81b55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c753567-2156-43dc-bc8d-5cae6bf81b55" (UID: "1c753567-2156-43dc-bc8d-5cae6bf81b55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:15:38.671662 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.671568    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c753567-2156-43dc-bc8d-5cae6bf81b55-kube-api-access-fhh7v" (OuterVolumeSpecName: "kube-api-access-fhh7v") pod "1c753567-2156-43dc-bc8d-5cae6bf81b55" (UID: "1c753567-2156-43dc-bc8d-5cae6bf81b55"). InnerVolumeSpecName "kube-api-access-fhh7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:15:38.762404 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.762315    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhh7v\" (UniqueName: \"kubernetes.io/projected/1c753567-2156-43dc-bc8d-5cae6bf81b55-kube-api-access-fhh7v\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:38.762404 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.762355    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c753567-2156-43dc-bc8d-5cae6bf81b55-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:38.762404 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:38.762370    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c753567-2156-43dc-bc8d-5cae6bf81b55-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:39.174098 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:39.174057    2561 generic.go:356] "Generic (PLEG): container finished" podID="1c753567-2156-43dc-bc8d-5cae6bf81b55" containerID="12b07c6abc8ef1dc35433c1bb69e66a8c4da583ecd86ab72c4e9433276c217f7" exitCode=0
Sep 10 20:15:39.174312 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:39.174107    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq" event={"ID":"1c753567-2156-43dc-bc8d-5cae6bf81b55","Type":"ContainerDied","Data":"12b07c6abc8ef1dc35433c1bb69e66a8c4da583ecd86ab72c4e9433276c217f7"}
Sep 10 20:15:39.174312 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:39.174131    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"
Sep 10 20:15:39.174312 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:39.174150    2561 scope.go:117] "RemoveContainer" containerID="12b07c6abc8ef1dc35433c1bb69e66a8c4da583ecd86ab72c4e9433276c217f7"
Sep 10 20:15:39.174312 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:39.174139    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq" event={"ID":"1c753567-2156-43dc-bc8d-5cae6bf81b55","Type":"ContainerDied","Data":"ff072bddc1a831d6e502e6b039941ca4caa115ed6f04f0960d2300002ca1ffca"}
Sep 10 20:15:39.183430 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:39.183407    2561 scope.go:117] "RemoveContainer" containerID="12b07c6abc8ef1dc35433c1bb69e66a8c4da583ecd86ab72c4e9433276c217f7"
Sep 10 20:15:39.183753 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:15:39.183734    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b07c6abc8ef1dc35433c1bb69e66a8c4da583ecd86ab72c4e9433276c217f7\": container with ID starting with 12b07c6abc8ef1dc35433c1bb69e66a8c4da583ecd86ab72c4e9433276c217f7 not found: ID does not exist" containerID="12b07c6abc8ef1dc35433c1bb69e66a8c4da583ecd86ab72c4e9433276c217f7"
Sep 10 20:15:39.183822 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:39.183761    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b07c6abc8ef1dc35433c1bb69e66a8c4da583ecd86ab72c4e9433276c217f7"} err="failed to get container status \"12b07c6abc8ef1dc35433c1bb69e66a8c4da583ecd86ab72c4e9433276c217f7\": rpc error: code = NotFound desc = could not find container \"12b07c6abc8ef1dc35433c1bb69e66a8c4da583ecd86ab72c4e9433276c217f7\": container with ID starting with 12b07c6abc8ef1dc35433c1bb69e66a8c4da583ecd86ab72c4e9433276c217f7 not found: ID does not exist"
Sep 10 20:15:39.197560 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:39.197515    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"]
Sep 10 20:15:39.203262 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:39.203236    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e88b3e4f/e2e-lifecycle-model-e88b3e4f-kserve-5884fc6d67-kjkpq"]
Sep 10 20:15:39.404588 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:39.404547    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c753567-2156-43dc-bc8d-5cae6bf81b55" path="/var/lib/kubelet/pods/1c753567-2156-43dc-bc8d-5cae6bf81b55/volumes"
Sep 10 20:15:39.405074 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:39.405047    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b4a09cb-18d5-46e5-8942-e0fbfe816a19" path="/var/lib/kubelet/pods/8b4a09cb-18d5-46e5-8942-e0fbfe816a19/volumes"
Sep 10 20:15:42.914813 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:42.914777    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:43.693124 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:43.693095    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:47.163404 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.163365    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"]
Sep 10 20:15:47.164232 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.164205    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c753567-2156-43dc-bc8d-5cae6bf81b55" containerName="main"
Sep 10 20:15:47.164232 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.164233    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c753567-2156-43dc-bc8d-5cae6bf81b55" containerName="main"
Sep 10 20:15:47.164452 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.164246    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b4a09cb-18d5-46e5-8942-e0fbfe816a19" containerName="test-curl-7751-e66a4c"
Sep 10 20:15:47.164452 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.164255    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b4a09cb-18d5-46e5-8942-e0fbfe816a19" containerName="test-curl-7751-e66a4c"
Sep 10 20:15:47.164737 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.164701    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b4a09cb-18d5-46e5-8942-e0fbfe816a19" containerName="test-curl-7751-e66a4c"
Sep 10 20:15:47.164831 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.164746    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c753567-2156-43dc-bc8d-5cae6bf81b55" containerName="main"
Sep 10 20:15:47.168920 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.168895    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.172908 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.172889    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-591d7c\"/\"sub-test-model-591d7c-kserve-self-signed-certs\""
Sep 10 20:15:47.178661 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.178641    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"]
Sep 10 20:15:47.245516 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.245488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kdpp\" (UniqueName: \"kubernetes.io/projected/5267c06b-2207-4af9-aad8-24c49d6e1b13-kube-api-access-4kdpp\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.245710 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.245549    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-home\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.245710 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.245587    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-tmp-dir\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.245710 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.245645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-dshm\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.245710 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.245689    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5267c06b-2207-4af9-aad8-24c49d6e1b13-tls-certs\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.245864 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.245717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-model-cache\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.257183 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.257153    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:47.346469 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.346431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kdpp\" (UniqueName: \"kubernetes.io/projected/5267c06b-2207-4af9-aad8-24c49d6e1b13-kube-api-access-4kdpp\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.346705 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.346483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-home\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.346705 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.346509    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-tmp-dir\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.346705 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.346567    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-dshm\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.346705 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.346601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5267c06b-2207-4af9-aad8-24c49d6e1b13-tls-certs\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.346705 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.346637    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-model-cache\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.346980 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.346955    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-home\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.347019 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.346990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-tmp-dir\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.347057 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.347032    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-model-cache\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.349055 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.349031    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-dshm\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.349317 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.349298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5267c06b-2207-4af9-aad8-24c49d6e1b13-tls-certs\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.356036 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.356018    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kdpp\" (UniqueName: \"kubernetes.io/projected/5267c06b-2207-4af9-aad8-24c49d6e1b13-kube-api-access-4kdpp\") pod \"sub-test-model-591d7c-kserve-7bffb99585-bx5ls\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.480551 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.480413    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:47.757500 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:47.757476    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"]
Sep 10 20:15:47.759032 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:15:47.759005    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5267c06b_2207_4af9_aad8_24c49d6e1b13.slice/crio-7833cd28e305a4b045d5c0ea05586afe9b47abfbece80c02ee3c3fcaf444eb68 WatchSource:0}: Error finding container 7833cd28e305a4b045d5c0ea05586afe9b47abfbece80c02ee3c3fcaf444eb68: Status 404 returned error can't find the container with id 7833cd28e305a4b045d5c0ea05586afe9b47abfbece80c02ee3c3fcaf444eb68
Sep 10 20:15:48.230346 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:48.230313    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls" event={"ID":"5267c06b-2207-4af9-aad8-24c49d6e1b13","Type":"ContainerStarted","Data":"0311e161f9b70424fc833022e668e8c0bbf8065c784f341058eb8935c95fc005"}
Sep 10 20:15:48.230346 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:48.230354    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls" event={"ID":"5267c06b-2207-4af9-aad8-24c49d6e1b13","Type":"ContainerStarted","Data":"7833cd28e305a4b045d5c0ea05586afe9b47abfbece80c02ee3c3fcaf444eb68"}
Sep 10 20:15:48.251888 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:48.251838    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls" podStartSLOduration=1.251822823 podStartE2EDuration="1.251822823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:15:48.247972947 +0000 UTC m=+1289.298500936" watchObservedRunningTime="2026-09-10 20:15:48.251822823 +0000 UTC m=+1289.302350812"
Sep 10 20:15:49.777873 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:49.777832    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:49.841402 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:49.841357    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:50.110910 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.110818    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:50.151081 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.151045    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc" podUID="a22273f6-0690-49e9-82ad-931e9aa1659f" containerName="main" containerID="cri-o://81eb365b27f33270c5cbf7a5c29ee5f4a8f2eb05532bdae81e895bf224e65b58" gracePeriod=45
Sep 10 20:15:50.396678 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.396636    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:15:50.475270 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.475217    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-home\") pod \"a22273f6-0690-49e9-82ad-931e9aa1659f\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") "
Sep 10 20:15:50.475270 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.475273    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22273f6-0690-49e9-82ad-931e9aa1659f-tls-certs\") pod \"a22273f6-0690-49e9-82ad-931e9aa1659f\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") "
Sep 10 20:15:50.475571 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.475299    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9p4dp\" (UniqueName: \"kubernetes.io/projected/a22273f6-0690-49e9-82ad-931e9aa1659f-kube-api-access-9p4dp\") pod \"a22273f6-0690-49e9-82ad-931e9aa1659f\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") "
Sep 10 20:15:50.475571 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.475334    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-dshm\") pod \"a22273f6-0690-49e9-82ad-931e9aa1659f\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") "
Sep 10 20:15:50.475571 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.475377    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-tmp-dir\") pod \"a22273f6-0690-49e9-82ad-931e9aa1659f\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") "
Sep 10 20:15:50.475571 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.475423    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-model-cache\") pod \"a22273f6-0690-49e9-82ad-931e9aa1659f\" (UID: \"a22273f6-0690-49e9-82ad-931e9aa1659f\") "
Sep 10 20:15:50.475818 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.475602    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-home" (OuterVolumeSpecName: "home") pod "a22273f6-0690-49e9-82ad-931e9aa1659f" (UID: "a22273f6-0690-49e9-82ad-931e9aa1659f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:15:50.475818 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.475772    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a22273f6-0690-49e9-82ad-931e9aa1659f" (UID: "a22273f6-0690-49e9-82ad-931e9aa1659f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:15:50.475818 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.475796    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a22273f6-0690-49e9-82ad-931e9aa1659f" (UID: "a22273f6-0690-49e9-82ad-931e9aa1659f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:15:50.475948 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.475842    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:50.477806 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.477775    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-dshm" (OuterVolumeSpecName: "dshm") pod "a22273f6-0690-49e9-82ad-931e9aa1659f" (UID: "a22273f6-0690-49e9-82ad-931e9aa1659f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:15:50.477915 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.477847    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22273f6-0690-49e9-82ad-931e9aa1659f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a22273f6-0690-49e9-82ad-931e9aa1659f" (UID: "a22273f6-0690-49e9-82ad-931e9aa1659f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:15:50.477915 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.477884    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22273f6-0690-49e9-82ad-931e9aa1659f-kube-api-access-9p4dp" (OuterVolumeSpecName: "kube-api-access-9p4dp") pod "a22273f6-0690-49e9-82ad-931e9aa1659f" (UID: "a22273f6-0690-49e9-82ad-931e9aa1659f"). InnerVolumeSpecName "kube-api-access-9p4dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:15:50.576968 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.576934    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p4dp\" (UniqueName: \"kubernetes.io/projected/a22273f6-0690-49e9-82ad-931e9aa1659f-kube-api-access-9p4dp\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:50.576968 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.576964    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:50.576968 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.576975    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:50.577211 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.576983    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22273f6-0690-49e9-82ad-931e9aa1659f-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:50.577211 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:50.576991    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22273f6-0690-49e9-82ad-931e9aa1659f-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:51.182105 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.182033    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c" podUID="54c01746-d707-49ff-829f-0cfc1cdc03bb" containerName="main" containerID="cri-o://5088c81ab2bb419713d15a1aa573e79c8e7ffa9f13e0b6dc018dc80a9429585e" gracePeriod=45
Sep 10 20:15:51.249680 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.249646    2561 generic.go:356] "Generic (PLEG): container finished" podID="a22273f6-0690-49e9-82ad-931e9aa1659f" containerID="81eb365b27f33270c5cbf7a5c29ee5f4a8f2eb05532bdae81e895bf224e65b58" exitCode=0
Sep 10 20:15:51.249846 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.249706    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"
Sep 10 20:15:51.249846 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.249737    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc" event={"ID":"a22273f6-0690-49e9-82ad-931e9aa1659f","Type":"ContainerDied","Data":"81eb365b27f33270c5cbf7a5c29ee5f4a8f2eb05532bdae81e895bf224e65b58"}
Sep 10 20:15:51.249846 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.249788    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc" event={"ID":"a22273f6-0690-49e9-82ad-931e9aa1659f","Type":"ContainerDied","Data":"7a6e8bea49b14336bbe53f8018dcfaf9ba798e13f6e6a42d4abe7f9131702180"}
Sep 10 20:15:51.249846 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.249809    2561 scope.go:117] "RemoveContainer" containerID="81eb365b27f33270c5cbf7a5c29ee5f4a8f2eb05532bdae81e895bf224e65b58"
Sep 10 20:15:51.273935 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.273901    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"]
Sep 10 20:15:51.280278 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.280249    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/auth-test-model-591d7c-kserve-58f7b556c-c8kfc"]
Sep 10 20:15:51.294139 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.294122    2561 scope.go:117] "RemoveContainer" containerID="81eb365b27f33270c5cbf7a5c29ee5f4a8f2eb05532bdae81e895bf224e65b58"
Sep 10 20:15:51.294482 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:15:51.294458    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81eb365b27f33270c5cbf7a5c29ee5f4a8f2eb05532bdae81e895bf224e65b58\": container with ID starting with 81eb365b27f33270c5cbf7a5c29ee5f4a8f2eb05532bdae81e895bf224e65b58 not found: ID does not exist" containerID="81eb365b27f33270c5cbf7a5c29ee5f4a8f2eb05532bdae81e895bf224e65b58"
Sep 10 20:15:51.294576 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.294496    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81eb365b27f33270c5cbf7a5c29ee5f4a8f2eb05532bdae81e895bf224e65b58"} err="failed to get container status \"81eb365b27f33270c5cbf7a5c29ee5f4a8f2eb05532bdae81e895bf224e65b58\": rpc error: code = NotFound desc = could not find container \"81eb365b27f33270c5cbf7a5c29ee5f4a8f2eb05532bdae81e895bf224e65b58\": container with ID starting with 81eb365b27f33270c5cbf7a5c29ee5f4a8f2eb05532bdae81e895bf224e65b58 not found: ID does not exist"
Sep 10 20:15:51.349396 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.349359    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:51.407183 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.407150    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a22273f6-0690-49e9-82ad-931e9aa1659f" path="/var/lib/kubelet/pods/a22273f6-0690-49e9-82ad-931e9aa1659f/volumes"
Sep 10 20:15:51.441129 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.441059    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:15:51.452567 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.452523    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:51.554243 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.554210    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:51.585331 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.585296    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c01746-d707-49ff-829f-0cfc1cdc03bb-tls-certs\") pod \"54c01746-d707-49ff-829f-0cfc1cdc03bb\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") "
Sep 10 20:15:51.585499 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.585340    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-model-cache\") pod \"54c01746-d707-49ff-829f-0cfc1cdc03bb\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") "
Sep 10 20:15:51.585499 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.585388    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-home\") pod \"54c01746-d707-49ff-829f-0cfc1cdc03bb\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") "
Sep 10 20:15:51.585499 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.585479    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bbbx\" (UniqueName: \"kubernetes.io/projected/54c01746-d707-49ff-829f-0cfc1cdc03bb-kube-api-access-7bbbx\") pod \"54c01746-d707-49ff-829f-0cfc1cdc03bb\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") "
Sep 10 20:15:51.585666 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.585508    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-dshm\") pod \"54c01746-d707-49ff-829f-0cfc1cdc03bb\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") "
Sep 10 20:15:51.585666 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.585558    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-tmp-dir\") pod \"54c01746-d707-49ff-829f-0cfc1cdc03bb\" (UID: \"54c01746-d707-49ff-829f-0cfc1cdc03bb\") "
Sep 10 20:15:51.585771 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.585715    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "54c01746-d707-49ff-829f-0cfc1cdc03bb" (UID: "54c01746-d707-49ff-829f-0cfc1cdc03bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:15:51.585840 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.585760    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-home" (OuterVolumeSpecName: "home") pod "54c01746-d707-49ff-829f-0cfc1cdc03bb" (UID: "54c01746-d707-49ff-829f-0cfc1cdc03bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:15:51.585906 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.585878    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54c01746-d707-49ff-829f-0cfc1cdc03bb" (UID: "54c01746-d707-49ff-829f-0cfc1cdc03bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:15:51.586015 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.585991    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:51.586015 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.586017    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:51.586365 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.586030    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:51.587868 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.587845    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c01746-d707-49ff-829f-0cfc1cdc03bb-kube-api-access-7bbbx" (OuterVolumeSpecName: "kube-api-access-7bbbx") pod "54c01746-d707-49ff-829f-0cfc1cdc03bb" (UID: "54c01746-d707-49ff-829f-0cfc1cdc03bb"). InnerVolumeSpecName "kube-api-access-7bbbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:15:51.587963 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.587889    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54c01746-d707-49ff-829f-0cfc1cdc03bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54c01746-d707-49ff-829f-0cfc1cdc03bb" (UID: "54c01746-d707-49ff-829f-0cfc1cdc03bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:15:51.587963 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.587908    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-dshm" (OuterVolumeSpecName: "dshm") pod "54c01746-d707-49ff-829f-0cfc1cdc03bb" (UID: "54c01746-d707-49ff-829f-0cfc1cdc03bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:15:51.686518 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.686476    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bbbx\" (UniqueName: \"kubernetes.io/projected/54c01746-d707-49ff-829f-0cfc1cdc03bb-kube-api-access-7bbbx\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:51.686518 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.686510    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c01746-d707-49ff-829f-0cfc1cdc03bb-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:51.686518 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:51.686521    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c01746-d707-49ff-829f-0cfc1cdc03bb-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:15:52.193957 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:52.193925    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:52.259055 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:52.259022    2561 generic.go:356] "Generic (PLEG): container finished" podID="54c01746-d707-49ff-829f-0cfc1cdc03bb" containerID="5088c81ab2bb419713d15a1aa573e79c8e7ffa9f13e0b6dc018dc80a9429585e" exitCode=0
Sep 10 20:15:52.259217 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:52.259070    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"
Sep 10 20:15:52.259217 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:52.259072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c" event={"ID":"54c01746-d707-49ff-829f-0cfc1cdc03bb","Type":"ContainerDied","Data":"5088c81ab2bb419713d15a1aa573e79c8e7ffa9f13e0b6dc018dc80a9429585e"}
Sep 10 20:15:52.259217 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:52.259112    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c" event={"ID":"54c01746-d707-49ff-829f-0cfc1cdc03bb","Type":"ContainerDied","Data":"10db60deed7a7205ca87087779fa4fad903fa7ee72d2d8bb3605379de8e8112a"}
Sep 10 20:15:52.259217 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:52.259134    2561 scope.go:117] "RemoveContainer" containerID="5088c81ab2bb419713d15a1aa573e79c8e7ffa9f13e0b6dc018dc80a9429585e"
Sep 10 20:15:52.268522 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:52.268497    2561 scope.go:117] "RemoveContainer" containerID="5088c81ab2bb419713d15a1aa573e79c8e7ffa9f13e0b6dc018dc80a9429585e"
Sep 10 20:15:52.268823 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:15:52.268804    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5088c81ab2bb419713d15a1aa573e79c8e7ffa9f13e0b6dc018dc80a9429585e\": container with ID starting with 5088c81ab2bb419713d15a1aa573e79c8e7ffa9f13e0b6dc018dc80a9429585e not found: ID does not exist" containerID="5088c81ab2bb419713d15a1aa573e79c8e7ffa9f13e0b6dc018dc80a9429585e"
Sep 10 20:15:52.268892 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:52.268831    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5088c81ab2bb419713d15a1aa573e79c8e7ffa9f13e0b6dc018dc80a9429585e"} err="failed to get container status \"5088c81ab2bb419713d15a1aa573e79c8e7ffa9f13e0b6dc018dc80a9429585e\": rpc error: code = NotFound desc = could not find container \"5088c81ab2bb419713d15a1aa573e79c8e7ffa9f13e0b6dc018dc80a9429585e\": container with ID starting with 5088c81ab2bb419713d15a1aa573e79c8e7ffa9f13e0b6dc018dc80a9429585e not found: ID does not exist"
Sep 10 20:15:52.282431 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:52.282400    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"]
Sep 10 20:15:52.286897 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:52.286850    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:52.289597 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:52.289576    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/auth-test-model-6550b4-kserve-76d494c88c-4hh5c"]
Sep 10 20:15:52.365996 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:52.365966    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:53.404697 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:53.404666    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54c01746-d707-49ff-829f-0cfc1cdc03bb" path="/var/lib/kubelet/pods/54c01746-d707-49ff-829f-0cfc1cdc03bb/volumes"
Sep 10 20:15:53.652187 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:53.652128    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:57.481358 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:57.481317    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:57.481842 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:57.481381    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:57.494176 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:57.494148    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:58.304802 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:58.304772    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:15:59.678143 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:59.678105    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:15:59.749273 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:15:59.749246    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:00.340128 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:00.340094    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:06.215514 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:06.215478    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:06.279038 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:06.278988    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:06.370204 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:06.370169    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:06.470271 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:06.470185    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:07.840837 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.840804    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"]
Sep 10 20:16:07.841211 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.841166    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22273f6-0690-49e9-82ad-931e9aa1659f" containerName="main"
Sep 10 20:16:07.841211 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.841177    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22273f6-0690-49e9-82ad-931e9aa1659f" containerName="main"
Sep 10 20:16:07.841211 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.841186    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c01746-d707-49ff-829f-0cfc1cdc03bb" containerName="main"
Sep 10 20:16:07.841211 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.841191    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c01746-d707-49ff-829f-0cfc1cdc03bb" containerName="main"
Sep 10 20:16:07.841358 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.841265    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22273f6-0690-49e9-82ad-931e9aa1659f" containerName="main"
Sep 10 20:16:07.841358 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.841274    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c01746-d707-49ff-829f-0cfc1cdc03bb" containerName="main"
Sep 10 20:16:07.845636 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.845618    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:07.848475 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.848453    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6550b4\"/\"default-dockercfg-fnx6v\""
Sep 10 20:16:07.849849 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.849830    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6550b4\"/\"kube-root-ca.crt\""
Sep 10 20:16:07.849958 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.849830    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6550b4\"/\"sub-test-model-6550b4-kserve-self-signed-certs\""
Sep 10 20:16:07.849958 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.849865    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6550b4\"/\"openshift-service-ca.crt\""
Sep 10 20:16:07.859564 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.859542    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"]
Sep 10 20:16:07.872577 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.872542    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:07.941988 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.941951    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-home\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:07.942200 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.942007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-dshm\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:07.942200 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.942098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz2q6\" (UniqueName: \"kubernetes.io/projected/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-kube-api-access-qz2q6\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:07.942200 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.942144    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-tmp-dir\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:07.942334 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.942206    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-tls-certs\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:07.942334 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:07.942255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-model-cache\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:08.043705 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:08.043664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz2q6\" (UniqueName: \"kubernetes.io/projected/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-kube-api-access-qz2q6\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:08.043915 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:08.043715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-tmp-dir\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:08.043915 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:08.043752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-tls-certs\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:08.043915 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:08.043811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-model-cache\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:08.043915 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:08.043855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-home\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:08.043915 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:08.043917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-dshm\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:08.044189 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:08.044149    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-tmp-dir\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:08.044256 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:08.044220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-model-cache\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:08.044256 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:08.044237    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-home\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:08.046360 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:08.046341    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-dshm\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:08.046489 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:08.046473    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-tls-certs\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:08.054043 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:08.054021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz2q6\" (UniqueName: \"kubernetes.io/projected/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-kube-api-access-qz2q6\") pod \"sub-test-model-6550b4-kserve-5c7c97d667-vk78n\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:08.155476 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:08.155369    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:08.434943 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:08.434914    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"]
Sep 10 20:16:08.436687 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:16:08.436662    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c5f2f2c_62b9_49f9_95b3_301ab4c2e470.slice/crio-f87fee7ae93b6d4af614dfde3820544aab30f81fdd1c4d19efd383be1315936c WatchSource:0}: Error finding container f87fee7ae93b6d4af614dfde3820544aab30f81fdd1c4d19efd383be1315936c: Status 404 returned error can't find the container with id f87fee7ae93b6d4af614dfde3820544aab30f81fdd1c4d19efd383be1315936c
Sep 10 20:16:09.354900 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:09.354863    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n" event={"ID":"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470","Type":"ContainerStarted","Data":"19d6de3a0a1d694958fdcfd7b2aeff797634046c29994347d555a9cce95f1a70"}
Sep 10 20:16:09.354900 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:09.354909    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n" event={"ID":"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470","Type":"ContainerStarted","Data":"f87fee7ae93b6d4af614dfde3820544aab30f81fdd1c4d19efd383be1315936c"}
Sep 10 20:16:09.375077 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:09.375016    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n" podStartSLOduration=2.375001934 podStartE2EDuration="2.375001934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:16:09.372852124 +0000 UTC m=+1310.423380115" watchObservedRunningTime="2026-09-10 20:16:09.375001934 +0000 UTC m=+1310.425529930"
Sep 10 20:16:16.915797 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:16.915763    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:17.155952 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:17.155921    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:18.156024 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:18.155981    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:18.156604 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:18.156057    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:18.169265 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:18.169236    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:18.415508 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:18.415414    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:16:38.129967 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:38.129927    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:43.433293 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:43.433204    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:44.040478 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:44.040445    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:44.139318 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:44.139284    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:44.386078 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:44.385999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:44.710900 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:44.710812    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:44.995117 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:44.995029    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:48.083498 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:48.083452    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:48.176426 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:48.176396    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:49.207490 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.207450    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"]
Sep 10 20:16:49.211369 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.211347    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:49.213770 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.213747    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bd3eab-serving-cert\""
Sep 10 20:16:49.226044 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.226012    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"]
Sep 10 20:16:49.242845 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.242813    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:49.327083 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.327049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ac1103-6728-4810-9f60-259fb82a575b-maas-api-tls\") pod \"maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:49.327083 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.327082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ac1103-6728-4810-9f60-259fb82a575b-openshift-service-ca\") pod \"maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:49.327304 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.327115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjz2d\" (UniqueName: \"kubernetes.io/projected/07ac1103-6728-4810-9f60-259fb82a575b-kube-api-access-rjz2d\") pod \"maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:49.327304 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.327221    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07ac1103-6728-4810-9f60-259fb82a575b-metrics-certs\") pod \"maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:49.345723 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.345695    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:49.428164 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.428125    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07ac1103-6728-4810-9f60-259fb82a575b-metrics-certs\") pod \"maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:49.428364 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.428260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ac1103-6728-4810-9f60-259fb82a575b-maas-api-tls\") pod \"maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:49.428364 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.428285    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ac1103-6728-4810-9f60-259fb82a575b-openshift-service-ca\") pod \"maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:49.428364 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.428330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjz2d\" (UniqueName: \"kubernetes.io/projected/07ac1103-6728-4810-9f60-259fb82a575b-kube-api-access-rjz2d\") pod \"maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:49.429017 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.428986    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ac1103-6728-4810-9f60-259fb82a575b-openshift-service-ca\") pod \"maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:49.430803 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.430782    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ac1103-6728-4810-9f60-259fb82a575b-maas-api-tls\") pod \"maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:49.430912 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.430888    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07ac1103-6728-4810-9f60-259fb82a575b-metrics-certs\") pod \"maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:49.436382 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.436364    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjz2d\" (UniqueName: \"kubernetes.io/projected/07ac1103-6728-4810-9f60-259fb82a575b-kube-api-access-rjz2d\") pod \"maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:49.444096 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.444073    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:49.521602 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.521569    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:49.722770 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.722705    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:49.800802 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.800752    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"]
Sep 10 20:16:49.801494 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.801442    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:16:49.802015 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:16:49.801991    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ac1103_6728_4810_9f60_259fb82a575b.slice/crio-0e4f388d54c02bc980c8f7baaf0823a34c8eeec18276454d75b89118c598f95c WatchSource:0}: Error finding container 0e4f388d54c02bc980c8f7baaf0823a34c8eeec18276454d75b89118c598f95c: Status 404 returned error can't find the container with id 0e4f388d54c02bc980c8f7baaf0823a34c8eeec18276454d75b89118c598f95c
Sep 10 20:16:49.803439 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:49.803421    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:16:50.578963 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:50.578924    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm" event={"ID":"07ac1103-6728-4810-9f60-259fb82a575b","Type":"ContainerStarted","Data":"50c31556bd7078d95ac7015d0a57b1c93ba712f20331f22c41435a32a8cff064"}
Sep 10 20:16:50.579362 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:50.578981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm" event={"ID":"07ac1103-6728-4810-9f60-259fb82a575b","Type":"ContainerStarted","Data":"0e4f388d54c02bc980c8f7baaf0823a34c8eeec18276454d75b89118c598f95c"}
Sep 10 20:16:50.579362 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:50.579135    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:50.597513 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:50.597415    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm" podStartSLOduration=1.06850676 podStartE2EDuration="1.59739858s" totalImagesPullingTime="528.89182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:16:50.596714878 +0000 UTC m=+1351.647242868" watchObservedRunningTime="2026-09-10 20:16:50.59739858 +0000 UTC m=+1351.647926614"
Sep 10 20:16:56.590462 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:56.590432    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:16:59.246631 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:59.246592    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7751-eabb7d"]
Sep 10 20:16:59.250284 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:59.250266    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-eabb7d"
Sep 10 20:16:59.252874 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:59.252850    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zcp8m\""
Sep 10 20:16:59.255754 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:59.255730    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7751-eabb7d"]
Sep 10 20:16:59.317968 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:59.317933    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26mfl\" (UniqueName: \"kubernetes.io/projected/fc81e76d-f705-4955-b6da-485088b0bf6a-kube-api-access-26mfl\") pod \"test-curl-7751-eabb7d\" (UID: \"fc81e76d-f705-4955-b6da-485088b0bf6a\") " pod="opendatahub/test-curl-7751-eabb7d"
Sep 10 20:16:59.418724 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:59.418695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26mfl\" (UniqueName: \"kubernetes.io/projected/fc81e76d-f705-4955-b6da-485088b0bf6a-kube-api-access-26mfl\") pod \"test-curl-7751-eabb7d\" (UID: \"fc81e76d-f705-4955-b6da-485088b0bf6a\") " pod="opendatahub/test-curl-7751-eabb7d"
Sep 10 20:16:59.426983 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:59.426951    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26mfl\" (UniqueName: \"kubernetes.io/projected/fc81e76d-f705-4955-b6da-485088b0bf6a-kube-api-access-26mfl\") pod \"test-curl-7751-eabb7d\" (UID: \"fc81e76d-f705-4955-b6da-485088b0bf6a\") " pod="opendatahub/test-curl-7751-eabb7d"
Sep 10 20:16:59.561632 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:59.561599    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-eabb7d"
Sep 10 20:16:59.835779 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:16:59.835737    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7751-eabb7d"]
Sep 10 20:16:59.837331 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:16:59.837302    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc81e76d_f705_4955_b6da_485088b0bf6a.slice/crio-5e538e286c97fa07193a7c2c290c63796743f41d1d7b63b8b30195e32b41f74a WatchSource:0}: Error finding container 5e538e286c97fa07193a7c2c290c63796743f41d1d7b63b8b30195e32b41f74a: Status 404 returned error can't find the container with id 5e538e286c97fa07193a7c2c290c63796743f41d1d7b63b8b30195e32b41f74a
Sep 10 20:17:00.642952 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:00.642908    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7751-eabb7d" event={"ID":"fc81e76d-f705-4955-b6da-485088b0bf6a","Type":"ContainerStarted","Data":"cdb69cf3e9e05ebe34a7c1a7481e52997081646bcd10597cadd51a5486aa0d4b"}
Sep 10 20:17:00.643394 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:00.642965    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7751-eabb7d" event={"ID":"fc81e76d-f705-4955-b6da-485088b0bf6a","Type":"ContainerStarted","Data":"5e538e286c97fa07193a7c2c290c63796743f41d1d7b63b8b30195e32b41f74a"}
Sep 10 20:17:00.658690 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:00.658642    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7751-eabb7d" podStartSLOduration=1.6586251 podStartE2EDuration="1.6586251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:17:00.658373996 +0000 UTC m=+1361.708902004" watchObservedRunningTime="2026-09-10 20:17:00.6586251 +0000 UTC m=+1361.709153088"
Sep 10 20:17:01.291583 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:01.291546    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7751-eabb7d"]
Sep 10 20:17:01.296849 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:01.296819    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7751-eabb7d"]
Sep 10 20:17:01.425196 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:01.425109    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7751-8fa7ee"]
Sep 10 20:17:01.425563 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:01.425549    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc81e76d-f705-4955-b6da-485088b0bf6a" containerName="test-curl-7751-eabb7d"
Sep 10 20:17:01.425615 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:01.425565    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc81e76d-f705-4955-b6da-485088b0bf6a" containerName="test-curl-7751-eabb7d"
Sep 10 20:17:01.425653 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:01.425648    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc81e76d-f705-4955-b6da-485088b0bf6a" containerName="test-curl-7751-eabb7d"
Sep 10 20:17:01.428916 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:01.428899    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-8fa7ee"
Sep 10 20:17:01.434967 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:01.434940    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7751-8fa7ee"]
Sep 10 20:17:01.538139 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:01.538098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkmsx\" (UniqueName: \"kubernetes.io/projected/f6b702bc-1d91-49bc-86ad-b85ca8e1f35a-kube-api-access-fkmsx\") pod \"test-curl-7751-8fa7ee\" (UID: \"f6b702bc-1d91-49bc-86ad-b85ca8e1f35a\") " pod="opendatahub/test-curl-7751-8fa7ee"
Sep 10 20:17:01.639746 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:01.639712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkmsx\" (UniqueName: \"kubernetes.io/projected/f6b702bc-1d91-49bc-86ad-b85ca8e1f35a-kube-api-access-fkmsx\") pod \"test-curl-7751-8fa7ee\" (UID: \"f6b702bc-1d91-49bc-86ad-b85ca8e1f35a\") " pod="opendatahub/test-curl-7751-8fa7ee"
Sep 10 20:17:01.648622 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:17:01.648592    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7751-eabb7d" container="test-curl-7751-eabb7d"
Sep 10 20:17:01.648975 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:01.648803    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkmsx\" (UniqueName: \"kubernetes.io/projected/f6b702bc-1d91-49bc-86ad-b85ca8e1f35a-kube-api-access-fkmsx\") pod \"test-curl-7751-8fa7ee\" (UID: \"f6b702bc-1d91-49bc-86ad-b85ca8e1f35a\") " pod="opendatahub/test-curl-7751-8fa7ee"
Sep 10 20:17:01.741033 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:01.740938    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-8fa7ee"
Sep 10 20:17:02.012276 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:02.012251    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7751-8fa7ee"]
Sep 10 20:17:02.014427 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:17:02.014393    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6b702bc_1d91_49bc_86ad_b85ca8e1f35a.slice/crio-c48bfc48c9d00aec0db5402fcb08c34181d05a429c9e8120496051d804453f48 WatchSource:0}: Error finding container c48bfc48c9d00aec0db5402fcb08c34181d05a429c9e8120496051d804453f48: Status 404 returned error can't find the container with id c48bfc48c9d00aec0db5402fcb08c34181d05a429c9e8120496051d804453f48
Sep 10 20:17:02.656223 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:02.656186    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7751-8fa7ee" event={"ID":"f6b702bc-1d91-49bc-86ad-b85ca8e1f35a","Type":"ContainerStarted","Data":"0ef5377066b71ddbc2d2383b65ffd2240d6e1fec62bf8d05beeec656ef191b58"}
Sep 10 20:17:02.656223 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:02.656231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7751-8fa7ee" event={"ID":"f6b702bc-1d91-49bc-86ad-b85ca8e1f35a","Type":"ContainerStarted","Data":"c48bfc48c9d00aec0db5402fcb08c34181d05a429c9e8120496051d804453f48"}
Sep 10 20:17:02.656728 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:02.656258    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7751-eabb7d" podUID="fc81e76d-f705-4955-b6da-485088b0bf6a" containerName="test-curl-7751-eabb7d" containerID="cri-o://cdb69cf3e9e05ebe34a7c1a7481e52997081646bcd10597cadd51a5486aa0d4b" gracePeriod=2
Sep 10 20:17:02.672854 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:02.672802    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7751-8fa7ee" podStartSLOduration=1.672786521 podStartE2EDuration="1.672786521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:17:02.66959288 +0000 UTC m=+1363.720120871" watchObservedRunningTime="2026-09-10 20:17:02.672786521 +0000 UTC m=+1363.723314508"
Sep 10 20:17:03.296684 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:03.296646    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7751-8fa7ee"]
Sep 10 20:17:03.300669 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:03.300640    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7751-8fa7ee"]
Sep 10 20:17:03.666400 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:17:03.666371    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7751-8fa7ee" container="test-curl-7751-8fa7ee"
Sep 10 20:17:04.671991 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:04.671936    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7751-8fa7ee" podUID="f6b702bc-1d91-49bc-86ad-b85ca8e1f35a" containerName="test-curl-7751-8fa7ee" containerID="cri-o://0ef5377066b71ddbc2d2383b65ffd2240d6e1fec62bf8d05beeec656ef191b58" gracePeriod=2
Sep 10 20:17:04.842959 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:04.842932    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-eabb7d"
Sep 10 20:17:04.975724 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:04.975631    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26mfl\" (UniqueName: \"kubernetes.io/projected/fc81e76d-f705-4955-b6da-485088b0bf6a-kube-api-access-26mfl\") pod \"fc81e76d-f705-4955-b6da-485088b0bf6a\" (UID: \"fc81e76d-f705-4955-b6da-485088b0bf6a\") "
Sep 10 20:17:04.977977 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:04.977953    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc81e76d-f705-4955-b6da-485088b0bf6a-kube-api-access-26mfl" (OuterVolumeSpecName: "kube-api-access-26mfl") pod "fc81e76d-f705-4955-b6da-485088b0bf6a" (UID: "fc81e76d-f705-4955-b6da-485088b0bf6a"). InnerVolumeSpecName "kube-api-access-26mfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:17:05.077354 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:05.077319    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26mfl\" (UniqueName: \"kubernetes.io/projected/fc81e76d-f705-4955-b6da-485088b0bf6a-kube-api-access-26mfl\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:17:05.349279 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:05.349249    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"]
Sep 10 20:17:05.369453 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:05.369417    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:05.404238 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:05.404202    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc81e76d-f705-4955-b6da-485088b0bf6a" path="/var/lib/kubelet/pods/fc81e76d-f705-4955-b6da-485088b0bf6a/volumes"
Sep 10 20:17:05.678999 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:05.678915    2561 generic.go:356] "Generic (PLEG): container finished" podID="fc81e76d-f705-4955-b6da-485088b0bf6a" containerID="cdb69cf3e9e05ebe34a7c1a7481e52997081646bcd10597cadd51a5486aa0d4b" exitCode=137
Sep 10 20:17:05.678999 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:05.678958    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-eabb7d"
Sep 10 20:17:05.679441 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:05.679011    2561 scope.go:117] "RemoveContainer" containerID="cdb69cf3e9e05ebe34a7c1a7481e52997081646bcd10597cadd51a5486aa0d4b"
Sep 10 20:17:05.688372 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:05.688346    2561 scope.go:117] "RemoveContainer" containerID="cdb69cf3e9e05ebe34a7c1a7481e52997081646bcd10597cadd51a5486aa0d4b"
Sep 10 20:17:05.688746 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:17:05.688726    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdb69cf3e9e05ebe34a7c1a7481e52997081646bcd10597cadd51a5486aa0d4b\": container with ID starting with cdb69cf3e9e05ebe34a7c1a7481e52997081646bcd10597cadd51a5486aa0d4b not found: ID does not exist" containerID="cdb69cf3e9e05ebe34a7c1a7481e52997081646bcd10597cadd51a5486aa0d4b"
Sep 10 20:17:05.688811 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:05.688757    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdb69cf3e9e05ebe34a7c1a7481e52997081646bcd10597cadd51a5486aa0d4b"} err="failed to get container status \"cdb69cf3e9e05ebe34a7c1a7481e52997081646bcd10597cadd51a5486aa0d4b\": rpc error: code = NotFound desc = could not find container \"cdb69cf3e9e05ebe34a7c1a7481e52997081646bcd10597cadd51a5486aa0d4b\": container with ID starting with cdb69cf3e9e05ebe34a7c1a7481e52997081646bcd10597cadd51a5486aa0d4b not found: ID does not exist"
Sep 10 20:17:06.261407 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:06.261363    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"]
Sep 10 20:17:06.290096 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:06.290053    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:06.843466 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:06.843440    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-8fa7ee"
Sep 10 20:17:06.996347 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:06.996248    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkmsx\" (UniqueName: \"kubernetes.io/projected/f6b702bc-1d91-49bc-86ad-b85ca8e1f35a-kube-api-access-fkmsx\") pod \"f6b702bc-1d91-49bc-86ad-b85ca8e1f35a\" (UID: \"f6b702bc-1d91-49bc-86ad-b85ca8e1f35a\") "
Sep 10 20:17:06.998588 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:06.998561    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b702bc-1d91-49bc-86ad-b85ca8e1f35a-kube-api-access-fkmsx" (OuterVolumeSpecName: "kube-api-access-fkmsx") pod "f6b702bc-1d91-49bc-86ad-b85ca8e1f35a" (UID: "f6b702bc-1d91-49bc-86ad-b85ca8e1f35a"). InnerVolumeSpecName "kube-api-access-fkmsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:17:07.097561 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:07.097493    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkmsx\" (UniqueName: \"kubernetes.io/projected/f6b702bc-1d91-49bc-86ad-b85ca8e1f35a-kube-api-access-fkmsx\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:17:07.404784 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:07.404752    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6b702bc-1d91-49bc-86ad-b85ca8e1f35a" path="/var/lib/kubelet/pods/f6b702bc-1d91-49bc-86ad-b85ca8e1f35a/volumes"
Sep 10 20:17:07.692597 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:07.692490    2561 generic.go:356] "Generic (PLEG): container finished" podID="f6b702bc-1d91-49bc-86ad-b85ca8e1f35a" containerID="0ef5377066b71ddbc2d2383b65ffd2240d6e1fec62bf8d05beeec656ef191b58" exitCode=137
Sep 10 20:17:07.692597 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:07.692567    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-8fa7ee"
Sep 10 20:17:07.692597 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:07.692582    2561 scope.go:117] "RemoveContainer" containerID="0ef5377066b71ddbc2d2383b65ffd2240d6e1fec62bf8d05beeec656ef191b58"
Sep 10 20:17:07.701776 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:07.701758    2561 scope.go:117] "RemoveContainer" containerID="0ef5377066b71ddbc2d2383b65ffd2240d6e1fec62bf8d05beeec656ef191b58"
Sep 10 20:17:07.702050 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:17:07.702028    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef5377066b71ddbc2d2383b65ffd2240d6e1fec62bf8d05beeec656ef191b58\": container with ID starting with 0ef5377066b71ddbc2d2383b65ffd2240d6e1fec62bf8d05beeec656ef191b58 not found: ID does not exist" containerID="0ef5377066b71ddbc2d2383b65ffd2240d6e1fec62bf8d05beeec656ef191b58"
Sep 10 20:17:07.702124 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:07.702058    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef5377066b71ddbc2d2383b65ffd2240d6e1fec62bf8d05beeec656ef191b58"} err="failed to get container status \"0ef5377066b71ddbc2d2383b65ffd2240d6e1fec62bf8d05beeec656ef191b58\": rpc error: code = NotFound desc = could not find container \"0ef5377066b71ddbc2d2383b65ffd2240d6e1fec62bf8d05beeec656ef191b58\": container with ID starting with 0ef5377066b71ddbc2d2383b65ffd2240d6e1fec62bf8d05beeec656ef191b58 not found: ID does not exist"
Sep 10 20:17:12.856550 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:12.856492    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:13.145942 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:13.145838    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:13.619165 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:13.619123    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:14.001472 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:14.001370    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:14.218878 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:14.218835    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:16.827097 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:16.827067    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:16.924777 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:16.924743    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:17.931877 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:17.931840    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"]
Sep 10 20:17:17.932420 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:17.932398    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b702bc-1d91-49bc-86ad-b85ca8e1f35a" containerName="test-curl-7751-8fa7ee"
Sep 10 20:17:17.932500 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:17.932424    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b702bc-1d91-49bc-86ad-b85ca8e1f35a" containerName="test-curl-7751-8fa7ee"
Sep 10 20:17:17.932594 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:17.932582    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6b702bc-1d91-49bc-86ad-b85ca8e1f35a" containerName="test-curl-7751-8fa7ee"
Sep 10 20:17:17.935661 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:17.935634    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:17.939257 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:17.939233    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-84c19e-serving-cert\""
Sep 10 20:17:17.948980 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:17.948955    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"]
Sep 10 20:17:17.980061 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:17.980020    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:18.074311 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.074271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:18.102543 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.102492    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a622b16c-49fd-48d3-84c2-8ab36c776e67-openshift-service-ca\") pod \"maas-api-e2e-api-b-84c19e-665b559d97-mn46l\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:18.102723 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.102572    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a622b16c-49fd-48d3-84c2-8ab36c776e67-metrics-certs\") pod \"maas-api-e2e-api-b-84c19e-665b559d97-mn46l\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:18.102723 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.102600    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7bzz\" (UniqueName: \"kubernetes.io/projected/a622b16c-49fd-48d3-84c2-8ab36c776e67-kube-api-access-p7bzz\") pod \"maas-api-e2e-api-b-84c19e-665b559d97-mn46l\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:18.102723 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.102630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a622b16c-49fd-48d3-84c2-8ab36c776e67-maas-api-tls\") pod \"maas-api-e2e-api-b-84c19e-665b559d97-mn46l\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:18.203374 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.203269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a622b16c-49fd-48d3-84c2-8ab36c776e67-openshift-service-ca\") pod \"maas-api-e2e-api-b-84c19e-665b559d97-mn46l\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:18.203374 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.203326    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a622b16c-49fd-48d3-84c2-8ab36c776e67-metrics-certs\") pod \"maas-api-e2e-api-b-84c19e-665b559d97-mn46l\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:18.203374 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.203358    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7bzz\" (UniqueName: \"kubernetes.io/projected/a622b16c-49fd-48d3-84c2-8ab36c776e67-kube-api-access-p7bzz\") pod \"maas-api-e2e-api-b-84c19e-665b559d97-mn46l\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:18.203691 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.203397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a622b16c-49fd-48d3-84c2-8ab36c776e67-maas-api-tls\") pod \"maas-api-e2e-api-b-84c19e-665b559d97-mn46l\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:18.203991 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.203968    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a622b16c-49fd-48d3-84c2-8ab36c776e67-openshift-service-ca\") pod \"maas-api-e2e-api-b-84c19e-665b559d97-mn46l\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:18.206003 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.205975    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a622b16c-49fd-48d3-84c2-8ab36c776e67-metrics-certs\") pod \"maas-api-e2e-api-b-84c19e-665b559d97-mn46l\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:18.206136 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.206115    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a622b16c-49fd-48d3-84c2-8ab36c776e67-maas-api-tls\") pod \"maas-api-e2e-api-b-84c19e-665b559d97-mn46l\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:18.212906 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.212885    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7bzz\" (UniqueName: \"kubernetes.io/projected/a622b16c-49fd-48d3-84c2-8ab36c776e67-kube-api-access-p7bzz\") pod \"maas-api-e2e-api-b-84c19e-665b559d97-mn46l\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:18.248957 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.248919    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:18.498844 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.498755    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:18.528261 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.528229    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"]
Sep 10 20:17:18.529864 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:17:18.529840    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda622b16c_49fd_48d3_84c2_8ab36c776e67.slice/crio-2272a6ed38e786cc50b2b4c6f710b53d85c8c9fdc103da9c968f2f0325ac3704 WatchSource:0}: Error finding container 2272a6ed38e786cc50b2b4c6f710b53d85c8c9fdc103da9c968f2f0325ac3704: Status 404 returned error can't find the container with id 2272a6ed38e786cc50b2b4c6f710b53d85c8c9fdc103da9c968f2f0325ac3704
Sep 10 20:17:18.576427 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.576391    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:18.762791 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:18.762691    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l" event={"ID":"a622b16c-49fd-48d3-84c2-8ab36c776e67","Type":"ContainerStarted","Data":"2272a6ed38e786cc50b2b4c6f710b53d85c8c9fdc103da9c968f2f0325ac3704"}
Sep 10 20:17:19.769824 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:19.769788    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l" event={"ID":"a622b16c-49fd-48d3-84c2-8ab36c776e67","Type":"ContainerStarted","Data":"47b8fed1680fdf476890c439a4902afa22f8e1cba11f4d32b73fb5e06875df3c"}
Sep 10 20:17:19.770230 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:19.769860    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:19.789071 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:19.789013    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l" podStartSLOduration=2.497549666 podStartE2EDuration="2.7889971s" totalImagesPullingTime="291.447434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:17:19.786929764 +0000 UTC m=+1380.837457754" watchObservedRunningTime="2026-09-10 20:17:19.7889971 +0000 UTC m=+1380.839525089"
Sep 10 20:17:20.369703 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.369660    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls" podUID="5267c06b-2207-4af9-aad8-24c49d6e1b13" containerName="main" containerID="cri-o://0311e161f9b70424fc833022e668e8c0bbf8065c784f341058eb8935c95fc005" gracePeriod=45
Sep 10 20:17:20.626445 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.626384    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:17:20.729342 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.729306    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-home\") pod \"5267c06b-2207-4af9-aad8-24c49d6e1b13\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") "
Sep 10 20:17:20.729342 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.729339    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-tmp-dir\") pod \"5267c06b-2207-4af9-aad8-24c49d6e1b13\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") "
Sep 10 20:17:20.729659 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.729385    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-dshm\") pod \"5267c06b-2207-4af9-aad8-24c49d6e1b13\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") "
Sep 10 20:17:20.729659 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.729408    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5267c06b-2207-4af9-aad8-24c49d6e1b13-tls-certs\") pod \"5267c06b-2207-4af9-aad8-24c49d6e1b13\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") "
Sep 10 20:17:20.729659 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.729428    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-model-cache\") pod \"5267c06b-2207-4af9-aad8-24c49d6e1b13\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") "
Sep 10 20:17:20.729659 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.729463    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kdpp\" (UniqueName: \"kubernetes.io/projected/5267c06b-2207-4af9-aad8-24c49d6e1b13-kube-api-access-4kdpp\") pod \"5267c06b-2207-4af9-aad8-24c49d6e1b13\" (UID: \"5267c06b-2207-4af9-aad8-24c49d6e1b13\") "
Sep 10 20:17:20.729850 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.729652    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5267c06b-2207-4af9-aad8-24c49d6e1b13" (UID: "5267c06b-2207-4af9-aad8-24c49d6e1b13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:17:20.729850 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.729672    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-home" (OuterVolumeSpecName: "home") pod "5267c06b-2207-4af9-aad8-24c49d6e1b13" (UID: "5267c06b-2207-4af9-aad8-24c49d6e1b13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:17:20.729850 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.729757    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-model-cache" (OuterVolumeSpecName: "model-cache") pod "5267c06b-2207-4af9-aad8-24c49d6e1b13" (UID: "5267c06b-2207-4af9-aad8-24c49d6e1b13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:17:20.729985 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.729930    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:17:20.729985 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.729942    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:17:20.729985 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.729952    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:17:20.731878 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.731846    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-dshm" (OuterVolumeSpecName: "dshm") pod "5267c06b-2207-4af9-aad8-24c49d6e1b13" (UID: "5267c06b-2207-4af9-aad8-24c49d6e1b13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:17:20.731878 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.731870    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5267c06b-2207-4af9-aad8-24c49d6e1b13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5267c06b-2207-4af9-aad8-24c49d6e1b13" (UID: "5267c06b-2207-4af9-aad8-24c49d6e1b13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:17:20.732034 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.731911    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5267c06b-2207-4af9-aad8-24c49d6e1b13-kube-api-access-4kdpp" (OuterVolumeSpecName: "kube-api-access-4kdpp") pod "5267c06b-2207-4af9-aad8-24c49d6e1b13" (UID: "5267c06b-2207-4af9-aad8-24c49d6e1b13"). InnerVolumeSpecName "kube-api-access-4kdpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:17:20.776758 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.776722    2561 generic.go:356] "Generic (PLEG): container finished" podID="5267c06b-2207-4af9-aad8-24c49d6e1b13" containerID="0311e161f9b70424fc833022e668e8c0bbf8065c784f341058eb8935c95fc005" exitCode=0
Sep 10 20:17:20.777213 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.776771    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"
Sep 10 20:17:20.777213 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.776821    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls" event={"ID":"5267c06b-2207-4af9-aad8-24c49d6e1b13","Type":"ContainerDied","Data":"0311e161f9b70424fc833022e668e8c0bbf8065c784f341058eb8935c95fc005"}
Sep 10 20:17:20.777213 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.776871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls" event={"ID":"5267c06b-2207-4af9-aad8-24c49d6e1b13","Type":"ContainerDied","Data":"7833cd28e305a4b045d5c0ea05586afe9b47abfbece80c02ee3c3fcaf444eb68"}
Sep 10 20:17:20.777213 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.776893    2561 scope.go:117] "RemoveContainer" containerID="0311e161f9b70424fc833022e668e8c0bbf8065c784f341058eb8935c95fc005"
Sep 10 20:17:20.787141 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.787124    2561 scope.go:117] "RemoveContainer" containerID="0311e161f9b70424fc833022e668e8c0bbf8065c784f341058eb8935c95fc005"
Sep 10 20:17:20.787458 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:17:20.787439    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0311e161f9b70424fc833022e668e8c0bbf8065c784f341058eb8935c95fc005\": container with ID starting with 0311e161f9b70424fc833022e668e8c0bbf8065c784f341058eb8935c95fc005 not found: ID does not exist" containerID="0311e161f9b70424fc833022e668e8c0bbf8065c784f341058eb8935c95fc005"
Sep 10 20:17:20.787503 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.787468    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0311e161f9b70424fc833022e668e8c0bbf8065c784f341058eb8935c95fc005"} err="failed to get container status \"0311e161f9b70424fc833022e668e8c0bbf8065c784f341058eb8935c95fc005\": rpc error: code = NotFound desc = could not find container \"0311e161f9b70424fc833022e668e8c0bbf8065c784f341058eb8935c95fc005\": container with ID starting with 0311e161f9b70424fc833022e668e8c0bbf8065c784f341058eb8935c95fc005 not found: ID does not exist"
Sep 10 20:17:20.805266 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.805228    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"]
Sep 10 20:17:20.812584 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.812556    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/sub-test-model-591d7c-kserve-7bffb99585-bx5ls"]
Sep 10 20:17:20.831255 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.831225    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5267c06b-2207-4af9-aad8-24c49d6e1b13-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:17:20.831255 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.831251    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kdpp\" (UniqueName: \"kubernetes.io/projected/5267c06b-2207-4af9-aad8-24c49d6e1b13-kube-api-access-4kdpp\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:17:20.831255 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:20.831262    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5267c06b-2207-4af9-aad8-24c49d6e1b13-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:17:21.282130 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.282081    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n" podUID="7c5f2f2c-62b9-49f9-95b3-301ab4c2e470" containerName="main" containerID="cri-o://19d6de3a0a1d694958fdcfd7b2aeff797634046c29994347d555a9cce95f1a70" gracePeriod=45
Sep 10 20:17:21.405355 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.405325    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5267c06b-2207-4af9-aad8-24c49d6e1b13" path="/var/lib/kubelet/pods/5267c06b-2207-4af9-aad8-24c49d6e1b13/volumes"
Sep 10 20:17:21.529813 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.529789    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:17:21.639203 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.639105    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-tmp-dir\") pod \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") "
Sep 10 20:17:21.639203 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.639194    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-dshm\") pod \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") "
Sep 10 20:17:21.639452 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.639240    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz2q6\" (UniqueName: \"kubernetes.io/projected/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-kube-api-access-qz2q6\") pod \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") "
Sep 10 20:17:21.639452 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.639269    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-model-cache\") pod \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") "
Sep 10 20:17:21.639452 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.639286    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-home\") pod \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") "
Sep 10 20:17:21.639452 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.639314    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-tls-certs\") pod \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\" (UID: \"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470\") "
Sep 10 20:17:21.639452 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.639429    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c5f2f2c-62b9-49f9-95b3-301ab4c2e470" (UID: "7c5f2f2c-62b9-49f9-95b3-301ab4c2e470"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:17:21.639720 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.639552    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c5f2f2c-62b9-49f9-95b3-301ab4c2e470" (UID: "7c5f2f2c-62b9-49f9-95b3-301ab4c2e470"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:17:21.639720 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.639574    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-home" (OuterVolumeSpecName: "home") pod "7c5f2f2c-62b9-49f9-95b3-301ab4c2e470" (UID: "7c5f2f2c-62b9-49f9-95b3-301ab4c2e470"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:17:21.639793 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.639768    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:17:21.639793 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.639788    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:17:21.639882 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.639800    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:17:21.641637 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.641605    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-dshm" (OuterVolumeSpecName: "dshm") pod "7c5f2f2c-62b9-49f9-95b3-301ab4c2e470" (UID: "7c5f2f2c-62b9-49f9-95b3-301ab4c2e470"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:17:21.641757 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.641666    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-kube-api-access-qz2q6" (OuterVolumeSpecName: "kube-api-access-qz2q6") pod "7c5f2f2c-62b9-49f9-95b3-301ab4c2e470" (UID: "7c5f2f2c-62b9-49f9-95b3-301ab4c2e470"). InnerVolumeSpecName "kube-api-access-qz2q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:17:21.641757 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.641662    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c5f2f2c-62b9-49f9-95b3-301ab4c2e470" (UID: "7c5f2f2c-62b9-49f9-95b3-301ab4c2e470"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:17:21.741247 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.741193    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:17:21.741247 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.741239    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz2q6\" (UniqueName: \"kubernetes.io/projected/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-kube-api-access-qz2q6\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:17:21.741247 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.741251    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:17:21.784175 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.784140    2561 generic.go:356] "Generic (PLEG): container finished" podID="7c5f2f2c-62b9-49f9-95b3-301ab4c2e470" containerID="19d6de3a0a1d694958fdcfd7b2aeff797634046c29994347d555a9cce95f1a70" exitCode=0
Sep 10 20:17:21.784657 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.784187    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"
Sep 10 20:17:21.784657 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.784183    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n" event={"ID":"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470","Type":"ContainerDied","Data":"19d6de3a0a1d694958fdcfd7b2aeff797634046c29994347d555a9cce95f1a70"}
Sep 10 20:17:21.784657 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.784240    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n" event={"ID":"7c5f2f2c-62b9-49f9-95b3-301ab4c2e470","Type":"ContainerDied","Data":"f87fee7ae93b6d4af614dfde3820544aab30f81fdd1c4d19efd383be1315936c"}
Sep 10 20:17:21.784657 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.784268    2561 scope.go:117] "RemoveContainer" containerID="19d6de3a0a1d694958fdcfd7b2aeff797634046c29994347d555a9cce95f1a70"
Sep 10 20:17:21.794460 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.794442    2561 scope.go:117] "RemoveContainer" containerID="19d6de3a0a1d694958fdcfd7b2aeff797634046c29994347d555a9cce95f1a70"
Sep 10 20:17:21.794757 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:17:21.794741    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19d6de3a0a1d694958fdcfd7b2aeff797634046c29994347d555a9cce95f1a70\": container with ID starting with 19d6de3a0a1d694958fdcfd7b2aeff797634046c29994347d555a9cce95f1a70 not found: ID does not exist" containerID="19d6de3a0a1d694958fdcfd7b2aeff797634046c29994347d555a9cce95f1a70"
Sep 10 20:17:21.794811 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.794764    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19d6de3a0a1d694958fdcfd7b2aeff797634046c29994347d555a9cce95f1a70"} err="failed to get container status \"19d6de3a0a1d694958fdcfd7b2aeff797634046c29994347d555a9cce95f1a70\": rpc error: code = NotFound desc = could not find container \"19d6de3a0a1d694958fdcfd7b2aeff797634046c29994347d555a9cce95f1a70\": container with ID starting with 19d6de3a0a1d694958fdcfd7b2aeff797634046c29994347d555a9cce95f1a70 not found: ID does not exist"
Sep 10 20:17:21.806046 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.806017    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"]
Sep 10 20:17:21.810216 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:21.810192    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/sub-test-model-6550b4-kserve-5c7c97d667-vk78n"]
Sep 10 20:17:23.404609 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:23.404562    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c5f2f2c-62b9-49f9-95b3-301ab4c2e470" path="/var/lib/kubelet/pods/7c5f2f2c-62b9-49f9-95b3-301ab4c2e470/volumes"
Sep 10 20:17:25.451363 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.451327    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"]
Sep 10 20:17:25.451778 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.451723    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5267c06b-2207-4af9-aad8-24c49d6e1b13" containerName="main"
Sep 10 20:17:25.451778 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.451735    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5267c06b-2207-4af9-aad8-24c49d6e1b13" containerName="main"
Sep 10 20:17:25.451778 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.451752    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5f2f2c-62b9-49f9-95b3-301ab4c2e470" containerName="main"
Sep 10 20:17:25.451778 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.451757    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5f2f2c-62b9-49f9-95b3-301ab4c2e470" containerName="main"
Sep 10 20:17:25.451910 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.451833    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5267c06b-2207-4af9-aad8-24c49d6e1b13" containerName="main"
Sep 10 20:17:25.451910 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.451841    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c5f2f2c-62b9-49f9-95b3-301ab4c2e470" containerName="main"
Sep 10 20:17:25.456337 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.456322    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.458917 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.458893    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-591d7c\"/\"openshift-service-ca.crt\""
Sep 10 20:17:25.459036 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.459016    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-591d7c\"/\"default-dockercfg-fkpb5\""
Sep 10 20:17:25.459875 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.459859    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-591d7c\"/\"kube-root-ca.crt\""
Sep 10 20:17:25.459929 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.459912    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-591d7c\"/\"rate-test-model-591d7c-kserve-self-signed-certs\""
Sep 10 20:17:25.465202 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.465177    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"]
Sep 10 20:17:25.474398 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.474370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-home\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.474547 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.474441    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ttb8\" (UniqueName: \"kubernetes.io/projected/c19f095e-f39a-4c7f-88a3-9994715281b7-kube-api-access-2ttb8\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.474547 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.474484    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19f095e-f39a-4c7f-88a3-9994715281b7-tls-certs\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.474639 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.474603    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-tmp-dir\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.474684 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.474672    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-dshm\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.474723 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.474695    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-model-cache\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.548338 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.548300    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:25.576130 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.576095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-dshm\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.576336 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.576139    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-model-cache\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.576336 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.576166    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-home\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.576336 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.576201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ttb8\" (UniqueName: \"kubernetes.io/projected/c19f095e-f39a-4c7f-88a3-9994715281b7-kube-api-access-2ttb8\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.576336 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.576233    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19f095e-f39a-4c7f-88a3-9994715281b7-tls-certs\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.576336 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.576281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-tmp-dir\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.576639 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.576596    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-model-cache\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.576681 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.576637    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-home\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.576792 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.576773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-tmp-dir\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.578742 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.578721    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-dshm\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.578900 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.578883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19f095e-f39a-4c7f-88a3-9994715281b7-tls-certs\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.585696 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.585676    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ttb8\" (UniqueName: \"kubernetes.io/projected/c19f095e-f39a-4c7f-88a3-9994715281b7-kube-api-access-2ttb8\") pod \"rate-test-model-591d7c-kserve-9f885fd7-qn2ms\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") " pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.766938 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.766852    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:25.783897 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:25.783867    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:17:26.049316 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:26.049287    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"]
Sep 10 20:17:26.050696 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:17:26.050664    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc19f095e_f39a_4c7f_88a3_9994715281b7.slice/crio-77aaeca0240ac75abc9975c6eaf408cb3915a716765306708f3de2eb77f89001 WatchSource:0}: Error finding container 77aaeca0240ac75abc9975c6eaf408cb3915a716765306708f3de2eb77f89001: Status 404 returned error can't find the container with id 77aaeca0240ac75abc9975c6eaf408cb3915a716765306708f3de2eb77f89001
Sep 10 20:17:26.815312 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:26.815279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms" event={"ID":"c19f095e-f39a-4c7f-88a3-9994715281b7","Type":"ContainerStarted","Data":"1e73170fdbd8a6a5fb3d47526c5619900bad8aa94d3e190adb9ada92c267c3eb"}
Sep 10 20:17:26.815312 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:26.815314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms" event={"ID":"c19f095e-f39a-4c7f-88a3-9994715281b7","Type":"ContainerStarted","Data":"77aaeca0240ac75abc9975c6eaf408cb3915a716765306708f3de2eb77f89001"}
Sep 10 20:17:26.832762 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:26.832712    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms" podStartSLOduration=1.832694927 podStartE2EDuration="1.832694927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:17:26.83122307 +0000 UTC m=+1387.881751059" watchObservedRunningTime="2026-09-10 20:17:26.832694927 +0000 UTC m=+1387.883222948"
Sep 10 20:17:35.767203 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:35.767151    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:35.767674 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:35.767225    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:35.780191 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:35.780165    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:35.874321 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:35.874286    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:17:41.825446 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:41.825409    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:44.969322 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:44.969278    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:49.652906 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.652874    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"]
Sep 10 20:17:49.656138 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.656118    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.659714 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.659685    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6550b4\"/\"kube-root-ca.crt\""
Sep 10 20:17:49.659714 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.659687    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6550b4\"/\"default-dockercfg-fnx6v\""
Sep 10 20:17:49.659931 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.659687    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6550b4\"/\"openshift-service-ca.crt\""
Sep 10 20:17:49.659931 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.659777    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6550b4\"/\"rate-test-model-6550b4-kserve-self-signed-certs\""
Sep 10 20:17:49.666174 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.666153    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"]
Sep 10 20:17:49.674714 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.674687    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:17:49.807940 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.807910    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-model-cache\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.808119 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.807972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pqk8\" (UniqueName: \"kubernetes.io/projected/7ebdb737-6a5c-4ec0-b049-984ac44b3223-kube-api-access-7pqk8\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.808119 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.808048    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ebdb737-6a5c-4ec0-b049-984ac44b3223-tls-certs\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.808119 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.808083    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-dshm\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.808119 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.808107    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-tmp-dir\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.808338 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.808166    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-home\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.908822 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.908714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-tmp-dir\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.908822 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.908777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-home\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.909077 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.908848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-model-cache\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.909077 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.908919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pqk8\" (UniqueName: \"kubernetes.io/projected/7ebdb737-6a5c-4ec0-b049-984ac44b3223-kube-api-access-7pqk8\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.909077 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.908972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ebdb737-6a5c-4ec0-b049-984ac44b3223-tls-certs\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.909077 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.909010    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-dshm\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.909314 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.909178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-tmp-dir\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.909314 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.909214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-home\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.909314 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.909263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-model-cache\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.911440 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.911420    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-dshm\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.911632 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.911614    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ebdb737-6a5c-4ec0-b049-984ac44b3223-tls-certs\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.918014 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.917985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pqk8\" (UniqueName: \"kubernetes.io/projected/7ebdb737-6a5c-4ec0-b049-984ac44b3223-kube-api-access-7pqk8\") pod \"rate-test-model-6550b4-kserve-5b486b5f47-c2x86\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") " pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:49.967638 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:49.967591    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:50.244169 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:50.244147    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"]
Sep 10 20:17:50.245840 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:17:50.245800    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ebdb737_6a5c_4ec0_b049_984ac44b3223.slice/crio-6fd808386baaaec06bb5cf187dce971834e962b8df742cf18505e7dc0a694ff7 WatchSource:0}: Error finding container 6fd808386baaaec06bb5cf187dce971834e962b8df742cf18505e7dc0a694ff7: Status 404 returned error can't find the container with id 6fd808386baaaec06bb5cf187dce971834e962b8df742cf18505e7dc0a694ff7
Sep 10 20:17:50.946035 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:50.945999    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86" event={"ID":"7ebdb737-6a5c-4ec0-b049-984ac44b3223","Type":"ContainerStarted","Data":"fb6dd7800ec088128439a2132b772d01a2793f0f17a3841a0dc32fa8af64089a"}
Sep 10 20:17:50.946458 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:50.946050    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86" event={"ID":"7ebdb737-6a5c-4ec0-b049-984ac44b3223","Type":"ContainerStarted","Data":"6fd808386baaaec06bb5cf187dce971834e962b8df742cf18505e7dc0a694ff7"}
Sep 10 20:17:50.963586 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:50.963507    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86" podStartSLOduration=1.96348817 podStartE2EDuration="1.96348817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:17:50.962497444 +0000 UTC m=+1412.013025434" watchObservedRunningTime="2026-09-10 20:17:50.96348817 +0000 UTC m=+1412.014016158"
Sep 10 20:17:59.968421 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:59.968382    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:59.968942 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:59.968458    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:17:59.981155 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:17:59.981129    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:18:00.015178 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.015144    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:18:00.336691 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.336660    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"]
Sep 10 20:18:00.336960 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.336927    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm" podUID="07ac1103-6728-4810-9f60-259fb82a575b" containerName="maas-api" containerID="cri-o://50c31556bd7078d95ac7015d0a57b1c93ba712f20331f22c41435a32a8cff064" gracePeriod=30
Sep 10 20:18:00.366048 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.366015    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:00.467961 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.467920    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:00.561341 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.561311    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:00.591788 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.591726    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:18:00.610582 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.610557    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjz2d\" (UniqueName: \"kubernetes.io/projected/07ac1103-6728-4810-9f60-259fb82a575b-kube-api-access-rjz2d\") pod \"07ac1103-6728-4810-9f60-259fb82a575b\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") "
Sep 10 20:18:00.610740 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.610615    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ac1103-6728-4810-9f60-259fb82a575b-maas-api-tls\") pod \"07ac1103-6728-4810-9f60-259fb82a575b\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") "
Sep 10 20:18:00.610740 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.610665    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ac1103-6728-4810-9f60-259fb82a575b-openshift-service-ca\") pod \"07ac1103-6728-4810-9f60-259fb82a575b\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") "
Sep 10 20:18:00.610740 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.610691    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07ac1103-6728-4810-9f60-259fb82a575b-metrics-certs\") pod \"07ac1103-6728-4810-9f60-259fb82a575b\" (UID: \"07ac1103-6728-4810-9f60-259fb82a575b\") "
Sep 10 20:18:00.611090 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.611061    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ac1103-6728-4810-9f60-259fb82a575b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07ac1103-6728-4810-9f60-259fb82a575b" (UID: "07ac1103-6728-4810-9f60-259fb82a575b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:18:00.613336 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.613308    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ac1103-6728-4810-9f60-259fb82a575b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "07ac1103-6728-4810-9f60-259fb82a575b" (UID: "07ac1103-6728-4810-9f60-259fb82a575b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:18:00.613336 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.613321    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ac1103-6728-4810-9f60-259fb82a575b-kube-api-access-rjz2d" (OuterVolumeSpecName: "kube-api-access-rjz2d") pod "07ac1103-6728-4810-9f60-259fb82a575b" (UID: "07ac1103-6728-4810-9f60-259fb82a575b"). InnerVolumeSpecName "kube-api-access-rjz2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:18:00.613826 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.613803    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ac1103-6728-4810-9f60-259fb82a575b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07ac1103-6728-4810-9f60-259fb82a575b" (UID: "07ac1103-6728-4810-9f60-259fb82a575b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:18:00.640953 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.640918    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:00.711345 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.711307    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ac1103-6728-4810-9f60-259fb82a575b-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:00.711345 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.711339    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ac1103-6728-4810-9f60-259fb82a575b-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:00.711345 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.711349    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07ac1103-6728-4810-9f60-259fb82a575b-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:00.711631 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:00.711357    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjz2d\" (UniqueName: \"kubernetes.io/projected/07ac1103-6728-4810-9f60-259fb82a575b-kube-api-access-rjz2d\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:01.011574 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:01.011450    2561 generic.go:356] "Generic (PLEG): container finished" podID="07ac1103-6728-4810-9f60-259fb82a575b" containerID="50c31556bd7078d95ac7015d0a57b1c93ba712f20331f22c41435a32a8cff064" exitCode=0
Sep 10 20:18:01.011574 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:01.011499    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"
Sep 10 20:18:01.011574 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:01.011561    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm" event={"ID":"07ac1103-6728-4810-9f60-259fb82a575b","Type":"ContainerDied","Data":"50c31556bd7078d95ac7015d0a57b1c93ba712f20331f22c41435a32a8cff064"}
Sep 10 20:18:01.012192 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:01.011625    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm" event={"ID":"07ac1103-6728-4810-9f60-259fb82a575b","Type":"ContainerDied","Data":"0e4f388d54c02bc980c8f7baaf0823a34c8eeec18276454d75b89118c598f95c"}
Sep 10 20:18:01.012192 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:01.011652    2561 scope.go:117] "RemoveContainer" containerID="50c31556bd7078d95ac7015d0a57b1c93ba712f20331f22c41435a32a8cff064"
Sep 10 20:18:01.021731 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:01.021712    2561 scope.go:117] "RemoveContainer" containerID="50c31556bd7078d95ac7015d0a57b1c93ba712f20331f22c41435a32a8cff064"
Sep 10 20:18:01.022010 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:18:01.021991    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c31556bd7078d95ac7015d0a57b1c93ba712f20331f22c41435a32a8cff064\": container with ID starting with 50c31556bd7078d95ac7015d0a57b1c93ba712f20331f22c41435a32a8cff064 not found: ID does not exist" containerID="50c31556bd7078d95ac7015d0a57b1c93ba712f20331f22c41435a32a8cff064"
Sep 10 20:18:01.022076 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:01.022022    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c31556bd7078d95ac7015d0a57b1c93ba712f20331f22c41435a32a8cff064"} err="failed to get container status \"50c31556bd7078d95ac7015d0a57b1c93ba712f20331f22c41435a32a8cff064\": rpc error: code = NotFound desc = could not find container \"50c31556bd7078d95ac7015d0a57b1c93ba712f20331f22c41435a32a8cff064\": container with ID starting with 50c31556bd7078d95ac7015d0a57b1c93ba712f20331f22c41435a32a8cff064 not found: ID does not exist"
Sep 10 20:18:01.023023 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:01.022998    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:01.047646 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:01.047621    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"]
Sep 10 20:18:01.053251 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:01.053227    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd3eab-847664ccd5-z4btm"]
Sep 10 20:18:01.404601 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:01.404566    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07ac1103-6728-4810-9f60-259fb82a575b" path="/var/lib/kubelet/pods/07ac1103-6728-4810-9f60-259fb82a575b/volumes"
Sep 10 20:18:01.624736 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:01.624698    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:07.099284 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:07.099251    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:07.199640 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:07.199594    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:07.504685 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:07.504589    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:07.823821 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:07.823787    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:23.140312 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.140277    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"]
Sep 10 20:18:23.140966 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.140578    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l" podUID="a622b16c-49fd-48d3-84c2-8ab36c776e67" containerName="maas-api" containerID="cri-o://47b8fed1680fdf476890c439a4902afa22f8e1cba11f4d32b73fb5e06875df3c" gracePeriod=30
Sep 10 20:18:23.190292 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.190257    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:23.289501 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.289472    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:23.386923 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.386898    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:18:23.409451 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.409359    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:23.469553 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.469500    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:23.530278 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.530245    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a622b16c-49fd-48d3-84c2-8ab36c776e67-maas-api-tls\") pod \"a622b16c-49fd-48d3-84c2-8ab36c776e67\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") "
Sep 10 20:18:23.530465 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.530291    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a622b16c-49fd-48d3-84c2-8ab36c776e67-openshift-service-ca\") pod \"a622b16c-49fd-48d3-84c2-8ab36c776e67\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") "
Sep 10 20:18:23.530465 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.530342    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a622b16c-49fd-48d3-84c2-8ab36c776e67-metrics-certs\") pod \"a622b16c-49fd-48d3-84c2-8ab36c776e67\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") "
Sep 10 20:18:23.530465 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.530382    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7bzz\" (UniqueName: \"kubernetes.io/projected/a622b16c-49fd-48d3-84c2-8ab36c776e67-kube-api-access-p7bzz\") pod \"a622b16c-49fd-48d3-84c2-8ab36c776e67\" (UID: \"a622b16c-49fd-48d3-84c2-8ab36c776e67\") "
Sep 10 20:18:23.530760 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.530731    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a622b16c-49fd-48d3-84c2-8ab36c776e67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a622b16c-49fd-48d3-84c2-8ab36c776e67" (UID: "a622b16c-49fd-48d3-84c2-8ab36c776e67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:18:23.530875 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.530841    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a622b16c-49fd-48d3-84c2-8ab36c776e67-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:23.532835 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.532803    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a622b16c-49fd-48d3-84c2-8ab36c776e67-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a622b16c-49fd-48d3-84c2-8ab36c776e67" (UID: "a622b16c-49fd-48d3-84c2-8ab36c776e67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:18:23.532940 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.532857    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a622b16c-49fd-48d3-84c2-8ab36c776e67-kube-api-access-p7bzz" (OuterVolumeSpecName: "kube-api-access-p7bzz") pod "a622b16c-49fd-48d3-84c2-8ab36c776e67" (UID: "a622b16c-49fd-48d3-84c2-8ab36c776e67"). InnerVolumeSpecName "kube-api-access-p7bzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:18:23.532940 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.532879    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a622b16c-49fd-48d3-84c2-8ab36c776e67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a622b16c-49fd-48d3-84c2-8ab36c776e67" (UID: "a622b16c-49fd-48d3-84c2-8ab36c776e67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:18:23.632340 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.632299    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a622b16c-49fd-48d3-84c2-8ab36c776e67-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:23.632340 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.632331    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a622b16c-49fd-48d3-84c2-8ab36c776e67-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:23.632340 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.632346    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7bzz\" (UniqueName: \"kubernetes.io/projected/a622b16c-49fd-48d3-84c2-8ab36c776e67-kube-api-access-p7bzz\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:23.715793 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:23.715683    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:24.142951 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:24.142915    2561 generic.go:356] "Generic (PLEG): container finished" podID="a622b16c-49fd-48d3-84c2-8ab36c776e67" containerID="47b8fed1680fdf476890c439a4902afa22f8e1cba11f4d32b73fb5e06875df3c" exitCode=0
Sep 10 20:18:24.143443 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:24.142965    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"
Sep 10 20:18:24.143443 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:24.142976    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l" event={"ID":"a622b16c-49fd-48d3-84c2-8ab36c776e67","Type":"ContainerDied","Data":"47b8fed1680fdf476890c439a4902afa22f8e1cba11f4d32b73fb5e06875df3c"}
Sep 10 20:18:24.143443 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:24.143017    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l" event={"ID":"a622b16c-49fd-48d3-84c2-8ab36c776e67","Type":"ContainerDied","Data":"2272a6ed38e786cc50b2b4c6f710b53d85c8c9fdc103da9c968f2f0325ac3704"}
Sep 10 20:18:24.143443 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:24.143038    2561 scope.go:117] "RemoveContainer" containerID="47b8fed1680fdf476890c439a4902afa22f8e1cba11f4d32b73fb5e06875df3c"
Sep 10 20:18:24.153243 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:24.153222    2561 scope.go:117] "RemoveContainer" containerID="47b8fed1680fdf476890c439a4902afa22f8e1cba11f4d32b73fb5e06875df3c"
Sep 10 20:18:24.153601 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:18:24.153575    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b8fed1680fdf476890c439a4902afa22f8e1cba11f4d32b73fb5e06875df3c\": container with ID starting with 47b8fed1680fdf476890c439a4902afa22f8e1cba11f4d32b73fb5e06875df3c not found: ID does not exist" containerID="47b8fed1680fdf476890c439a4902afa22f8e1cba11f4d32b73fb5e06875df3c"
Sep 10 20:18:24.153667 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:24.153611    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b8fed1680fdf476890c439a4902afa22f8e1cba11f4d32b73fb5e06875df3c"} err="failed to get container status \"47b8fed1680fdf476890c439a4902afa22f8e1cba11f4d32b73fb5e06875df3c\": rpc error: code = NotFound desc = could not find container \"47b8fed1680fdf476890c439a4902afa22f8e1cba11f4d32b73fb5e06875df3c\": container with ID starting with 47b8fed1680fdf476890c439a4902afa22f8e1cba11f4d32b73fb5e06875df3c not found: ID does not exist"
Sep 10 20:18:24.164988 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:24.164956    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"]
Sep 10 20:18:24.167373 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:24.167348    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84c19e-665b559d97-mn46l"]
Sep 10 20:18:25.404419 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:25.404385    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a622b16c-49fd-48d3-84c2-8ab36c776e67" path="/var/lib/kubelet/pods/a622b16c-49fd-48d3-84c2-8ab36c776e67/volumes"
Sep 10 20:18:29.597342 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:29.597305    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:29.658748 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:29.658714    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:29.851396 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:29.851305    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:30.439801 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:30.439771    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:31.159407 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:31.159372    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:31.945372 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:31.945337    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:40.652343 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:40.652306    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"]
Sep 10 20:18:40.652821 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:40.652549    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr" podUID="b0bae2ce-9f7d-49c5-bb89-691f723e34d1" containerName="maas-api" containerID="cri-o://c7a5aa173660fd4c3bab923de7cb839587183bd8e00c0d582cadefb3e1b9ec1a" gracePeriod=30
Sep 10 20:18:40.679768 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:40.679730    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:40.752505 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:40.752462    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:40.816996 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:40.816960    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:40.891965 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:40.891914    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:40.905059 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:40.904993    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:18:40.975877 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:40.975842    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:40.998521 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:40.998475    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq29m\" (UniqueName: \"kubernetes.io/projected/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-kube-api-access-tq29m\") pod \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") "
Sep 10 20:18:40.998740 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:40.998566    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-openshift-service-ca\") pod \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") "
Sep 10 20:18:40.998740 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:40.998620    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-metrics-certs\") pod \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") "
Sep 10 20:18:40.998740 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:40.998649    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-maas-api-tls\") pod \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\" (UID: \"b0bae2ce-9f7d-49c5-bb89-691f723e34d1\") "
Sep 10 20:18:40.999115 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:40.999078    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0bae2ce-9f7d-49c5-bb89-691f723e34d1" (UID: "b0bae2ce-9f7d-49c5-bb89-691f723e34d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:18:41.000985 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.000956    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b0bae2ce-9f7d-49c5-bb89-691f723e34d1" (UID: "b0bae2ce-9f7d-49c5-bb89-691f723e34d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:18:41.000985 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.000958    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-kube-api-access-tq29m" (OuterVolumeSpecName: "kube-api-access-tq29m") pod "b0bae2ce-9f7d-49c5-bb89-691f723e34d1" (UID: "b0bae2ce-9f7d-49c5-bb89-691f723e34d1"). InnerVolumeSpecName "kube-api-access-tq29m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:18:41.001222 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.001196    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0bae2ce-9f7d-49c5-bb89-691f723e34d1" (UID: "b0bae2ce-9f7d-49c5-bb89-691f723e34d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:18:41.099820 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.099766    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:41.099820 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.099808    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:41.099820 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.099822    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:41.100130 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.099835    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq29m\" (UniqueName: \"kubernetes.io/projected/b0bae2ce-9f7d-49c5-bb89-691f723e34d1-kube-api-access-tq29m\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:41.237705 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.237606    2561 generic.go:356] "Generic (PLEG): container finished" podID="b0bae2ce-9f7d-49c5-bb89-691f723e34d1" containerID="c7a5aa173660fd4c3bab923de7cb839587183bd8e00c0d582cadefb3e1b9ec1a" exitCode=0
Sep 10 20:18:41.237705 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.237671    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"
Sep 10 20:18:41.237896 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.237702    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr" event={"ID":"b0bae2ce-9f7d-49c5-bb89-691f723e34d1","Type":"ContainerDied","Data":"c7a5aa173660fd4c3bab923de7cb839587183bd8e00c0d582cadefb3e1b9ec1a"}
Sep 10 20:18:41.237896 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.237779    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr" event={"ID":"b0bae2ce-9f7d-49c5-bb89-691f723e34d1","Type":"ContainerDied","Data":"d836f11c195a198c50a0edf58dc7816b977f99830f76de8aee09a8299154d0a6"}
Sep 10 20:18:41.237896 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.237805    2561 scope.go:117] "RemoveContainer" containerID="c7a5aa173660fd4c3bab923de7cb839587183bd8e00c0d582cadefb3e1b9ec1a"
Sep 10 20:18:41.248172 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.248152    2561 scope.go:117] "RemoveContainer" containerID="c7a5aa173660fd4c3bab923de7cb839587183bd8e00c0d582cadefb3e1b9ec1a"
Sep 10 20:18:41.248625 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:18:41.248503    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a5aa173660fd4c3bab923de7cb839587183bd8e00c0d582cadefb3e1b9ec1a\": container with ID starting with c7a5aa173660fd4c3bab923de7cb839587183bd8e00c0d582cadefb3e1b9ec1a not found: ID does not exist" containerID="c7a5aa173660fd4c3bab923de7cb839587183bd8e00c0d582cadefb3e1b9ec1a"
Sep 10 20:18:41.248625 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.248555    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a5aa173660fd4c3bab923de7cb839587183bd8e00c0d582cadefb3e1b9ec1a"} err="failed to get container status \"c7a5aa173660fd4c3bab923de7cb839587183bd8e00c0d582cadefb3e1b9ec1a\": rpc error: code = NotFound desc = could not find container \"c7a5aa173660fd4c3bab923de7cb839587183bd8e00c0d582cadefb3e1b9ec1a\": container with ID starting with c7a5aa173660fd4c3bab923de7cb839587183bd8e00c0d582cadefb3e1b9ec1a not found: ID does not exist"
Sep 10 20:18:41.273040 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.273002    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"]
Sep 10 20:18:41.285756 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.285722    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-214b09-6678c6cdff-8zknr"]
Sep 10 20:18:41.289334 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.289303    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:41.404226 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:41.404189    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0bae2ce-9f7d-49c5-bb89-691f723e34d1" path="/var/lib/kubelet/pods/b0bae2ce-9f7d-49c5-bb89-691f723e34d1/volumes"
Sep 10 20:18:47.122843 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:47.122800    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:47.224276 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:47.224242    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:47.366173 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:47.366140    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:47.753627 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:47.753589    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"]
Sep 10 20:18:47.777281 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:47.777250    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:47.962498 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:47.962461    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:48.637970 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:48.637936    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"]
Sep 10 20:18:48.659104 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:48.659064    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:49.101476 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:49.101444    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:49.466917 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:49.466839    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:49.724399 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:49.724305    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:49.974019 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:49.973985    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:52.659896 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:52.659850    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:52.758416 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:52.758374    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:53.788891 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.788851    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"]
Sep 10 20:18:53.789419 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.789367    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0bae2ce-9f7d-49c5-bb89-691f723e34d1" containerName="maas-api"
Sep 10 20:18:53.789419 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.789385    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0bae2ce-9f7d-49c5-bb89-691f723e34d1" containerName="maas-api"
Sep 10 20:18:53.789419 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.789412    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07ac1103-6728-4810-9f60-259fb82a575b" containerName="maas-api"
Sep 10 20:18:53.789419 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.789419    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="07ac1103-6728-4810-9f60-259fb82a575b" containerName="maas-api"
Sep 10 20:18:53.789719 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.789435    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a622b16c-49fd-48d3-84c2-8ab36c776e67" containerName="maas-api"
Sep 10 20:18:53.789719 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.789444    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a622b16c-49fd-48d3-84c2-8ab36c776e67" containerName="maas-api"
Sep 10 20:18:53.789719 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.789512    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="07ac1103-6728-4810-9f60-259fb82a575b" containerName="maas-api"
Sep 10 20:18:53.789719 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.789520    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a622b16c-49fd-48d3-84c2-8ab36c776e67" containerName="maas-api"
Sep 10 20:18:53.789719 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.789547    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0bae2ce-9f7d-49c5-bb89-691f723e34d1" containerName="maas-api"
Sep 10 20:18:53.792697 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.792679    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:53.795920 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.795896    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-316fde-serving-cert\""
Sep 10 20:18:53.806656 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.806628    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"]
Sep 10 20:18:53.821582 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.821542    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-metrics-certs\") pod \"maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:53.821769 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.821618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:53.821769 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.821666    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:53.821769 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.821682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgfq\" (UniqueName: \"kubernetes.io/projected/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-kube-api-access-sxgfq\") pod \"maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:53.834126 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.834095    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:53.922644 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.922611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:53.922644 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.922646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgfq\" (UniqueName: \"kubernetes.io/projected/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-kube-api-access-sxgfq\") pod \"maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:53.922915 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.922715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-metrics-certs\") pod \"maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:53.922915 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:18:53.922760    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-serving-cert: secret "maas-api-e2e-ipp-a-316fde-serving-cert" not found
Sep 10 20:18:53.922915 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:18:53.922840    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-maas-api-tls podName:0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1 nodeName:}" failed. No retries permitted until 2026-09-10 20:18:54.422820726 +0000 UTC m=+1475.473348693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-maas-api-tls") pod "maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6" (UID: "0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1") : secret "maas-api-e2e-ipp-a-316fde-serving-cert" not found
Sep 10 20:18:53.922915 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.922865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:53.923548 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.923512    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:53.925200 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.925180    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-metrics-certs\") pod \"maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:53.932784 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.932761    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgfq\" (UniqueName: \"kubernetes.io/projected/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-kube-api-access-sxgfq\") pod \"maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:53.988220 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:53.988186    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:54.097403 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:54.097313    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:54.395572 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:54.395463    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:54.427987 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:54.427955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:54.430475 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:54.430451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:54.490871 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:54.490839    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:54.704349 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:54.704253    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:54.983902 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:54.983875    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"]
Sep 10 20:18:54.985441 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:18:54.985413    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd656c0_0e5a_4937_ac1e_fa0af2d0ddd1.slice/crio-54b562554de1ee5427da425e36e23eef8e2017e509f9f9811716646e78338efc WatchSource:0}: Error finding container 54b562554de1ee5427da425e36e23eef8e2017e509f9f9811716646e78338efc: Status 404 returned error can't find the container with id 54b562554de1ee5427da425e36e23eef8e2017e509f9f9811716646e78338efc
Sep 10 20:18:55.319538 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:55.319494    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6" event={"ID":"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1","Type":"ContainerStarted","Data":"54b562554de1ee5427da425e36e23eef8e2017e509f9f9811716646e78338efc"}
Sep 10 20:18:56.326219 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:56.326185    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6" event={"ID":"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1","Type":"ContainerStarted","Data":"a93395d8faa80e8dfc230ad95d52777dc50943d8740f71dad89abb9c0f98776d"}
Sep 10 20:18:56.326740 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:56.326330    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:18:56.343566 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:56.343498    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6" podStartSLOduration=2.904431464 podStartE2EDuration="3.343479687s" totalImagesPullingTime="439.048223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:18:56.342036428 +0000 UTC m=+1477.392564429" watchObservedRunningTime="2026-09-10 20:18:56.343479687 +0000 UTC m=+1477.394007676"
Sep 10 20:18:57.842833 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:57.842796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:58.400254 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.400226    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"]
Sep 10 20:18:58.400464 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.400441    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg" podUID="06eb9d90-f48b-4ea7-a353-bb274e0e43aa" containerName="maas-api" containerID="cri-o://30fe97bfdaf1a3edd1df6ca7f28a6b26c453d9e3afbc67423520b76d466fc0d9" gracePeriod=30
Sep 10 20:18:58.439642 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.439599    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:58.509134 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.509101    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:58.583219 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.583190    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:58.653062 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.652985    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:58.660144 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.660118    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:18:58.770742 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.770709    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-openshift-service-ca\") pod \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") "
Sep 10 20:18:58.770924 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.770761    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-maas-api-tls\") pod \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") "
Sep 10 20:18:58.770924 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.770794    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqh5r\" (UniqueName: \"kubernetes.io/projected/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-kube-api-access-mqh5r\") pod \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") "
Sep 10 20:18:58.770924 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.770844    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-metrics-certs\") pod \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\" (UID: \"06eb9d90-f48b-4ea7-a353-bb274e0e43aa\") "
Sep 10 20:18:58.771092 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.771039    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06eb9d90-f48b-4ea7-a353-bb274e0e43aa" (UID: "06eb9d90-f48b-4ea7-a353-bb274e0e43aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:18:58.773063 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.773032    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06eb9d90-f48b-4ea7-a353-bb274e0e43aa" (UID: "06eb9d90-f48b-4ea7-a353-bb274e0e43aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:18:58.773226 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.773202    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "06eb9d90-f48b-4ea7-a353-bb274e0e43aa" (UID: "06eb9d90-f48b-4ea7-a353-bb274e0e43aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:18:58.773274 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.773222    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-kube-api-access-mqh5r" (OuterVolumeSpecName: "kube-api-access-mqh5r") pod "06eb9d90-f48b-4ea7-a353-bb274e0e43aa" (UID: "06eb9d90-f48b-4ea7-a353-bb274e0e43aa"). InnerVolumeSpecName "kube-api-access-mqh5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:18:58.871890 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.871853    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:58.871890 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.871884    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:58.871890 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.871894    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqh5r\" (UniqueName: \"kubernetes.io/projected/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-kube-api-access-mqh5r\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:58.872332 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:58.871902    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06eb9d90-f48b-4ea7-a353-bb274e0e43aa-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:18:59.007135 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:59.004777    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:18:59.344182 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:59.344149    2561 generic.go:356] "Generic (PLEG): container finished" podID="06eb9d90-f48b-4ea7-a353-bb274e0e43aa" containerID="30fe97bfdaf1a3edd1df6ca7f28a6b26c453d9e3afbc67423520b76d466fc0d9" exitCode=0
Sep 10 20:18:59.344363 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:59.344199    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"
Sep 10 20:18:59.344363 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:59.344234    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg" event={"ID":"06eb9d90-f48b-4ea7-a353-bb274e0e43aa","Type":"ContainerDied","Data":"30fe97bfdaf1a3edd1df6ca7f28a6b26c453d9e3afbc67423520b76d466fc0d9"}
Sep 10 20:18:59.344363 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:59.344312    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg" event={"ID":"06eb9d90-f48b-4ea7-a353-bb274e0e43aa","Type":"ContainerDied","Data":"f2691b211287c86f49e852b7c83452a08f9e101a710ea44d2972732cec833abb"}
Sep 10 20:18:59.344363 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:59.344335    2561 scope.go:117] "RemoveContainer" containerID="30fe97bfdaf1a3edd1df6ca7f28a6b26c453d9e3afbc67423520b76d466fc0d9"
Sep 10 20:18:59.354562 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:59.354541    2561 scope.go:117] "RemoveContainer" containerID="30fe97bfdaf1a3edd1df6ca7f28a6b26c453d9e3afbc67423520b76d466fc0d9"
Sep 10 20:18:59.354866 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:18:59.354846    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30fe97bfdaf1a3edd1df6ca7f28a6b26c453d9e3afbc67423520b76d466fc0d9\": container with ID starting with 30fe97bfdaf1a3edd1df6ca7f28a6b26c453d9e3afbc67423520b76d466fc0d9 not found: ID does not exist" containerID="30fe97bfdaf1a3edd1df6ca7f28a6b26c453d9e3afbc67423520b76d466fc0d9"
Sep 10 20:18:59.354941 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:59.354879    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30fe97bfdaf1a3edd1df6ca7f28a6b26c453d9e3afbc67423520b76d466fc0d9"} err="failed to get container status \"30fe97bfdaf1a3edd1df6ca7f28a6b26c453d9e3afbc67423520b76d466fc0d9\": rpc error: code = NotFound desc = could not find container \"30fe97bfdaf1a3edd1df6ca7f28a6b26c453d9e3afbc67423520b76d466fc0d9\": container with ID starting with 30fe97bfdaf1a3edd1df6ca7f28a6b26c453d9e3afbc67423520b76d466fc0d9 not found: ID does not exist"
Sep 10 20:18:59.367348 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:59.367216    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"]
Sep 10 20:18:59.369948 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:59.369920    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ef723-6bb5c468c9-bcwtg"]
Sep 10 20:18:59.403909 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:18:59.403871    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06eb9d90-f48b-4ea7-a353-bb274e0e43aa" path="/var/lib/kubelet/pods/06eb9d90-f48b-4ea7-a353-bb274e0e43aa/volumes"
Sep 10 20:19:02.337265 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:02.337232    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:19:02.773884 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:02.773847    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms" podUID="c19f095e-f39a-4c7f-88a3-9994715281b7" containerName="main" containerID="cri-o://1e73170fdbd8a6a5fb3d47526c5619900bad8aa94d3e190adb9ada92c267c3eb" gracePeriod=45
Sep 10 20:19:03.033711 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.033651    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:19:03.114090 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.114046    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-home\") pod \"c19f095e-f39a-4c7f-88a3-9994715281b7\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") "
Sep 10 20:19:03.114288 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.114139    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-model-cache\") pod \"c19f095e-f39a-4c7f-88a3-9994715281b7\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") "
Sep 10 20:19:03.114288 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.114226    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-tmp-dir\") pod \"c19f095e-f39a-4c7f-88a3-9994715281b7\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") "
Sep 10 20:19:03.114288 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.114257    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19f095e-f39a-4c7f-88a3-9994715281b7-tls-certs\") pod \"c19f095e-f39a-4c7f-88a3-9994715281b7\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") "
Sep 10 20:19:03.114288 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.114279    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-dshm\") pod \"c19f095e-f39a-4c7f-88a3-9994715281b7\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") "
Sep 10 20:19:03.114487 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.114311    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ttb8\" (UniqueName: \"kubernetes.io/projected/c19f095e-f39a-4c7f-88a3-9994715281b7-kube-api-access-2ttb8\") pod \"c19f095e-f39a-4c7f-88a3-9994715281b7\" (UID: \"c19f095e-f39a-4c7f-88a3-9994715281b7\") "
Sep 10 20:19:03.114487 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.114351    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-home" (OuterVolumeSpecName: "home") pod "c19f095e-f39a-4c7f-88a3-9994715281b7" (UID: "c19f095e-f39a-4c7f-88a3-9994715281b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:19:03.114615 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.114476    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "c19f095e-f39a-4c7f-88a3-9994715281b7" (UID: "c19f095e-f39a-4c7f-88a3-9994715281b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:19:03.114615 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.114502    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c19f095e-f39a-4c7f-88a3-9994715281b7" (UID: "c19f095e-f39a-4c7f-88a3-9994715281b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:19:03.114735 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.114644    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:19:03.114735 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.114658    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:19:03.114735 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.114668    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:19:03.116746 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.116718    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19f095e-f39a-4c7f-88a3-9994715281b7-kube-api-access-2ttb8" (OuterVolumeSpecName: "kube-api-access-2ttb8") pod "c19f095e-f39a-4c7f-88a3-9994715281b7" (UID: "c19f095e-f39a-4c7f-88a3-9994715281b7"). InnerVolumeSpecName "kube-api-access-2ttb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:19:03.116858 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.116763    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-dshm" (OuterVolumeSpecName: "dshm") pod "c19f095e-f39a-4c7f-88a3-9994715281b7" (UID: "c19f095e-f39a-4c7f-88a3-9994715281b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:19:03.116858 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.116802    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19f095e-f39a-4c7f-88a3-9994715281b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c19f095e-f39a-4c7f-88a3-9994715281b7" (UID: "c19f095e-f39a-4c7f-88a3-9994715281b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:19:03.216036 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.215993    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19f095e-f39a-4c7f-88a3-9994715281b7-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:19:03.216036 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.216026    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19f095e-f39a-4c7f-88a3-9994715281b7-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:19:03.216036 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.216036    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ttb8\" (UniqueName: \"kubernetes.io/projected/c19f095e-f39a-4c7f-88a3-9994715281b7-kube-api-access-2ttb8\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:19:03.368450 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.368360    2561 generic.go:356] "Generic (PLEG): container finished" podID="c19f095e-f39a-4c7f-88a3-9994715281b7" containerID="1e73170fdbd8a6a5fb3d47526c5619900bad8aa94d3e190adb9ada92c267c3eb" exitCode=0
Sep 10 20:19:03.368450 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.368423    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"
Sep 10 20:19:03.368929 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.368449    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms" event={"ID":"c19f095e-f39a-4c7f-88a3-9994715281b7","Type":"ContainerDied","Data":"1e73170fdbd8a6a5fb3d47526c5619900bad8aa94d3e190adb9ada92c267c3eb"}
Sep 10 20:19:03.368929 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.368497    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms" event={"ID":"c19f095e-f39a-4c7f-88a3-9994715281b7","Type":"ContainerDied","Data":"77aaeca0240ac75abc9975c6eaf408cb3915a716765306708f3de2eb77f89001"}
Sep 10 20:19:03.368929 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.368519    2561 scope.go:117] "RemoveContainer" containerID="1e73170fdbd8a6a5fb3d47526c5619900bad8aa94d3e190adb9ada92c267c3eb"
Sep 10 20:19:03.378388 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.378369    2561 scope.go:117] "RemoveContainer" containerID="1e73170fdbd8a6a5fb3d47526c5619900bad8aa94d3e190adb9ada92c267c3eb"
Sep 10 20:19:03.378724 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:19:03.378707    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e73170fdbd8a6a5fb3d47526c5619900bad8aa94d3e190adb9ada92c267c3eb\": container with ID starting with 1e73170fdbd8a6a5fb3d47526c5619900bad8aa94d3e190adb9ada92c267c3eb not found: ID does not exist" containerID="1e73170fdbd8a6a5fb3d47526c5619900bad8aa94d3e190adb9ada92c267c3eb"
Sep 10 20:19:03.378783 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.378734    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e73170fdbd8a6a5fb3d47526c5619900bad8aa94d3e190adb9ada92c267c3eb"} err="failed to get container status \"1e73170fdbd8a6a5fb3d47526c5619900bad8aa94d3e190adb9ada92c267c3eb\": rpc error: code = NotFound desc = could not find container \"1e73170fdbd8a6a5fb3d47526c5619900bad8aa94d3e190adb9ada92c267c3eb\": container with ID starting with 1e73170fdbd8a6a5fb3d47526c5619900bad8aa94d3e190adb9ada92c267c3eb not found: ID does not exist"
Sep 10 20:19:03.390662 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.390639    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"]
Sep 10 20:19:03.396116 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.396081    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-591d7c/rate-test-model-591d7c-kserve-9f885fd7-qn2ms"]
Sep 10 20:19:03.404289 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.404263    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c19f095e-f39a-4c7f-88a3-9994715281b7" path="/var/lib/kubelet/pods/c19f095e-f39a-4c7f-88a3-9994715281b7/volumes"
Sep 10 20:19:03.658996 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.658903    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86" podUID="7ebdb737-6a5c-4ec0-b049-984ac44b3223" containerName="main" containerID="cri-o://fb6dd7800ec088128439a2132b772d01a2793f0f17a3841a0dc32fa8af64089a" gracePeriod=45
Sep 10 20:19:03.918287 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:03.918221    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:19:04.022042 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.021994    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-home\") pod \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") "
Sep 10 20:19:04.022253 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.022068    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pqk8\" (UniqueName: \"kubernetes.io/projected/7ebdb737-6a5c-4ec0-b049-984ac44b3223-kube-api-access-7pqk8\") pod \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") "
Sep 10 20:19:04.022253 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.022086    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-model-cache\") pod \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") "
Sep 10 20:19:04.022253 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.022167    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ebdb737-6a5c-4ec0-b049-984ac44b3223-tls-certs\") pod \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") "
Sep 10 20:19:04.022253 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.022185    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-dshm\") pod \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") "
Sep 10 20:19:04.022253 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.022204    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-tmp-dir\") pod \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\" (UID: \"7ebdb737-6a5c-4ec0-b049-984ac44b3223\") "
Sep 10 20:19:04.022519 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.022342    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-home" (OuterVolumeSpecName: "home") pod "7ebdb737-6a5c-4ec0-b049-984ac44b3223" (UID: "7ebdb737-6a5c-4ec0-b049-984ac44b3223"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:19:04.022519 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.022473    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:19:04.022519 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.022505    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ebdb737-6a5c-4ec0-b049-984ac44b3223" (UID: "7ebdb737-6a5c-4ec0-b049-984ac44b3223"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:19:04.022708 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.022556    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ebdb737-6a5c-4ec0-b049-984ac44b3223" (UID: "7ebdb737-6a5c-4ec0-b049-984ac44b3223"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:19:04.024445 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.024424    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebdb737-6a5c-4ec0-b049-984ac44b3223-kube-api-access-7pqk8" (OuterVolumeSpecName: "kube-api-access-7pqk8") pod "7ebdb737-6a5c-4ec0-b049-984ac44b3223" (UID: "7ebdb737-6a5c-4ec0-b049-984ac44b3223"). InnerVolumeSpecName "kube-api-access-7pqk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:19:04.024551 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.024450    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebdb737-6a5c-4ec0-b049-984ac44b3223-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ebdb737-6a5c-4ec0-b049-984ac44b3223" (UID: "7ebdb737-6a5c-4ec0-b049-984ac44b3223"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:19:04.024551 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.024510    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-dshm" (OuterVolumeSpecName: "dshm") pod "7ebdb737-6a5c-4ec0-b049-984ac44b3223" (UID: "7ebdb737-6a5c-4ec0-b049-984ac44b3223"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:19:04.123697 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.123650    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pqk8\" (UniqueName: \"kubernetes.io/projected/7ebdb737-6a5c-4ec0-b049-984ac44b3223-kube-api-access-7pqk8\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:19:04.123697 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.123692    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:19:04.123697 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.123701    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ebdb737-6a5c-4ec0-b049-984ac44b3223-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:19:04.123697 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.123709    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:19:04.123697 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.123717    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ebdb737-6a5c-4ec0-b049-984ac44b3223-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:19:04.376792 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.376755    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ebdb737-6a5c-4ec0-b049-984ac44b3223" containerID="fb6dd7800ec088128439a2132b772d01a2793f0f17a3841a0dc32fa8af64089a" exitCode=0
Sep 10 20:19:04.377282 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.376805    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"
Sep 10 20:19:04.377282 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.376844    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86" event={"ID":"7ebdb737-6a5c-4ec0-b049-984ac44b3223","Type":"ContainerDied","Data":"fb6dd7800ec088128439a2132b772d01a2793f0f17a3841a0dc32fa8af64089a"}
Sep 10 20:19:04.377282 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.376883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86" event={"ID":"7ebdb737-6a5c-4ec0-b049-984ac44b3223","Type":"ContainerDied","Data":"6fd808386baaaec06bb5cf187dce971834e962b8df742cf18505e7dc0a694ff7"}
Sep 10 20:19:04.377282 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.376901    2561 scope.go:117] "RemoveContainer" containerID="fb6dd7800ec088128439a2132b772d01a2793f0f17a3841a0dc32fa8af64089a"
Sep 10 20:19:04.386576 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.386558    2561 scope.go:117] "RemoveContainer" containerID="fb6dd7800ec088128439a2132b772d01a2793f0f17a3841a0dc32fa8af64089a"
Sep 10 20:19:04.386862 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:19:04.386844    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6dd7800ec088128439a2132b772d01a2793f0f17a3841a0dc32fa8af64089a\": container with ID starting with fb6dd7800ec088128439a2132b772d01a2793f0f17a3841a0dc32fa8af64089a not found: ID does not exist" containerID="fb6dd7800ec088128439a2132b772d01a2793f0f17a3841a0dc32fa8af64089a"
Sep 10 20:19:04.386928 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.386874    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6dd7800ec088128439a2132b772d01a2793f0f17a3841a0dc32fa8af64089a"} err="failed to get container status \"fb6dd7800ec088128439a2132b772d01a2793f0f17a3841a0dc32fa8af64089a\": rpc error: code = NotFound desc = could not find container \"fb6dd7800ec088128439a2132b772d01a2793f0f17a3841a0dc32fa8af64089a\": container with ID starting with fb6dd7800ec088128439a2132b772d01a2793f0f17a3841a0dc32fa8af64089a not found: ID does not exist"
Sep 10 20:19:04.399579 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.399555    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"]
Sep 10 20:19:04.404148 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.404115    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6550b4/rate-test-model-6550b4-kserve-5b486b5f47-c2x86"]
Sep 10 20:19:04.768483 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.768456    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:04.868579 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:04.868546    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:05.026846 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:05.026752    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:05.404253 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:05.404172    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ebdb737-6a5c-4ec0-b049-984ac44b3223" path="/var/lib/kubelet/pods/7ebdb737-6a5c-4ec0-b049-984ac44b3223/volumes"
Sep 10 20:19:14.954411 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:14.954374    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:15.253791 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:15.253681    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:17.614067 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:17.614036    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:18.279173 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:18.279141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:18.596302 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:18.596217    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:18.970857 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:18.970773    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:20.883888 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:20.883766    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 20:19:20.906633 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:20.890143    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 20:19:22.680131 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.680095    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:22.779959 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.779922    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:22.970876 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.970791    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"]
Sep 10 20:19:22.971273 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.971256    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19f095e-f39a-4c7f-88a3-9994715281b7" containerName="main"
Sep 10 20:19:22.971320 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.971277    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19f095e-f39a-4c7f-88a3-9994715281b7" containerName="main"
Sep 10 20:19:22.971320 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.971291    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06eb9d90-f48b-4ea7-a353-bb274e0e43aa" containerName="maas-api"
Sep 10 20:19:22.971320 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.971297    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="06eb9d90-f48b-4ea7-a353-bb274e0e43aa" containerName="maas-api"
Sep 10 20:19:22.971320 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.971313    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebdb737-6a5c-4ec0-b049-984ac44b3223" containerName="main"
Sep 10 20:19:22.971320 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.971318    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebdb737-6a5c-4ec0-b049-984ac44b3223" containerName="main"
Sep 10 20:19:22.971475 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.971370    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="06eb9d90-f48b-4ea7-a353-bb274e0e43aa" containerName="maas-api"
Sep 10 20:19:22.971475 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.971378    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19f095e-f39a-4c7f-88a3-9994715281b7" containerName="main"
Sep 10 20:19:22.971475 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.971384    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebdb737-6a5c-4ec0-b049-984ac44b3223" containerName="main"
Sep 10 20:19:22.974455 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.974434    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:22.977248 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.977227    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-82457a-serving-cert\""
Sep 10 20:19:22.984369 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:22.984345    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"]
Sep 10 20:19:23.003875 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.003850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-82457a-664df89fcf-6685d\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:23.004033 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.003914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-82457a-664df89fcf-6685d\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:23.004033 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.003949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-metrics-certs\") pod \"maas-api-e2e-ipp-b-82457a-664df89fcf-6685d\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:23.004033 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.003971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wgxr\" (UniqueName: \"kubernetes.io/projected/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-kube-api-access-8wgxr\") pod \"maas-api-e2e-ipp-b-82457a-664df89fcf-6685d\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:23.008296 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.008266    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:23.104447 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.104415    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-82457a-664df89fcf-6685d\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:23.104447 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.104451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-metrics-certs\") pod \"maas-api-e2e-ipp-b-82457a-664df89fcf-6685d\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:23.104740 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.104476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wgxr\" (UniqueName: \"kubernetes.io/projected/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-kube-api-access-8wgxr\") pod \"maas-api-e2e-ipp-b-82457a-664df89fcf-6685d\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:23.104740 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.104519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-82457a-664df89fcf-6685d\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:23.104740 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:19:23.104628    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-serving-cert: secret "maas-api-e2e-ipp-b-82457a-serving-cert" not found
Sep 10 20:19:23.104740 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:19:23.104694    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-maas-api-tls podName:c7b3683d-e5f9-4a37-bf25-f2693640d3a0 nodeName:}" failed. No retries permitted until 2026-09-10 20:19:23.604677396 +0000 UTC m=+1504.655205366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-maas-api-tls") pod "maas-api-e2e-ipp-b-82457a-664df89fcf-6685d" (UID: "c7b3683d-e5f9-4a37-bf25-f2693640d3a0") : secret "maas-api-e2e-ipp-b-82457a-serving-cert" not found
Sep 10 20:19:23.105149 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.105127    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-82457a-664df89fcf-6685d\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:23.107128 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.107104    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-metrics-certs\") pod \"maas-api-e2e-ipp-b-82457a-664df89fcf-6685d\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:23.113872 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.113848    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:23.114607 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.114589    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wgxr\" (UniqueName: \"kubernetes.io/projected/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-kube-api-access-8wgxr\") pod \"maas-api-e2e-ipp-b-82457a-664df89fcf-6685d\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:23.295730 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.295696    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:23.488593 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.488560    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:23.581628 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.581544    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:23.610058 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.610023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-82457a-664df89fcf-6685d\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:23.612648 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.612618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-82457a-664df89fcf-6685d\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:23.887764 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:23.887677    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:24.163630 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:24.163597    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"]
Sep 10 20:19:24.165094 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:19:24.165067    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b3683d_e5f9_4a37_bf25_f2693640d3a0.slice/crio-eba3aaf1db6bb9f0afba765236288cb6414c996823c0766f8fd3a543b10fb4ff WatchSource:0}: Error finding container eba3aaf1db6bb9f0afba765236288cb6414c996823c0766f8fd3a543b10fb4ff: Status 404 returned error can't find the container with id eba3aaf1db6bb9f0afba765236288cb6414c996823c0766f8fd3a543b10fb4ff
Sep 10 20:19:24.485515 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:24.485427    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d" event={"ID":"c7b3683d-e5f9-4a37-bf25-f2693640d3a0","Type":"ContainerStarted","Data":"eba3aaf1db6bb9f0afba765236288cb6414c996823c0766f8fd3a543b10fb4ff"}
Sep 10 20:19:25.493449 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:25.493407    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d" event={"ID":"c7b3683d-e5f9-4a37-bf25-f2693640d3a0","Type":"ContainerStarted","Data":"aae05741e5081926873c4cbd613d605485346962f00af1ac776072120ef2015b"}
Sep 10 20:19:25.493870 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:25.493493    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:25.513855 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:25.513802    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d" podStartSLOduration=3.037129264 podStartE2EDuration="3.513784954s" totalImagesPullingTime="476.65569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:19:25.510889749 +0000 UTC m=+1506.561417738" watchObservedRunningTime="2026-09-10 20:19:25.513784954 +0000 UTC m=+1506.564312942"
Sep 10 20:19:31.504939 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:31.504910    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:19:40.917284 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:40.917192    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:40.985667 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:40.985628    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:41.904730 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:41.904693    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:19:46.864019 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:19:46.863984    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:13.390986 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:13.390948    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:13.468995 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:13.468954    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:17.351918 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.351885    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"]
Sep 10 20:20:17.355777 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.355754    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.358356 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.358331    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-316fde\"/\"kube-root-ca.crt\""
Sep 10 20:20:17.358507 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.358331    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-316fde\"/\"openshift-service-ca.crt\""
Sep 10 20:20:17.359232 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.359210    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-316fde\"/\"default-dockercfg-fd88m\""
Sep 10 20:20:17.359337 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.359246    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-316fde\"/\"ipp-route-316fde-kserve-self-signed-certs\""
Sep 10 20:20:17.367513 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.367483    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"]
Sep 10 20:20:17.389358 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.389325    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:17.419806 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.419769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6df6j\" (UniqueName: \"kubernetes.io/projected/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-kube-api-access-6df6j\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.419980 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.419813    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-dshm\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.419980 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.419865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-model-cache\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.419980 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.419967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-home\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.420118 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.420072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-tmp-dir\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.420118 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.420108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-tls-certs\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.521470 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.521431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-dshm\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.521470 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.521475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-model-cache\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.521780 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.521519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-home\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.521780 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.521655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-tmp-dir\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.521780 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.521705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-tls-certs\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.521946 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.521817    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6df6j\" (UniqueName: \"kubernetes.io/projected/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-kube-api-access-6df6j\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.522024 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.521993    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-tmp-dir\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.522195 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.522166    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-home\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.522334 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.522252    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-model-cache\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.523916 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.523897    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-dshm\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.524294 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.524271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-tls-certs\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.530183 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.530157    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6df6j\" (UniqueName: \"kubernetes.io/projected/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-kube-api-access-6df6j\") pod \"ipp-route-316fde-kserve-6c974887-njhpc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") " pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.666458 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.666375    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:17.943832 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:17.943800    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"]
Sep 10 20:20:17.945369 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:20:17.945337    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb97d1a3_8b3d_4108_b264_9a1b6e0a05cc.slice/crio-862e2670846e1235a700221f9bffd61a5f0e45753bdfd7740eeb262e4085435d WatchSource:0}: Error finding container 862e2670846e1235a700221f9bffd61a5f0e45753bdfd7740eeb262e4085435d: Status 404 returned error can't find the container with id 862e2670846e1235a700221f9bffd61a5f0e45753bdfd7740eeb262e4085435d
Sep 10 20:20:18.782386 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:18.782347    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc" event={"ID":"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc","Type":"ContainerStarted","Data":"860c2f6835e60627f5c410d813d43a950a9e378d94f0abdfc717e7abd7cecce7"}
Sep 10 20:20:18.782386 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:18.782392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc" event={"ID":"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc","Type":"ContainerStarted","Data":"862e2670846e1235a700221f9bffd61a5f0e45753bdfd7740eeb262e4085435d"}
Sep 10 20:20:18.801240 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:18.801189    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc" podStartSLOduration=1.8011696430000002 podStartE2EDuration="1.801169643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:20:18.80075192 +0000 UTC m=+1559.851279923" watchObservedRunningTime="2026-09-10 20:20:18.801169643 +0000 UTC m=+1559.851697637"
Sep 10 20:20:27.667230 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:27.667191    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:27.667681 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:27.667257    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:27.680093 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:27.680066    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:27.843444 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:27.843411    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:20:30.700705 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:30.700668    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:30.797985 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:30.797948    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:43.630715 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:43.630683    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:43.946485 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:43.946397    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:44.669943 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:44.669912    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:47.419241 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:47.419199    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:47.518012 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:47.517976    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:48.707873 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.707838    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"]
Sep 10 20:20:48.712014 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.711995    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:48.714685 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.714664    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-fe603d-serving-cert\""
Sep 10 20:20:48.723367 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.723346    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"]
Sep 10 20:20:48.742419 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.742379    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:48.816395 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.816363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:48.816616 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.816411    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:48.816616 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.816436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxb6l\" (UniqueName: \"kubernetes.io/projected/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-kube-api-access-lxb6l\") pod \"maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:48.816616 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.816455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:48.824224 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.824184    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:48.917305 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.917268    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:48.917509 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.917336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:48.917509 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.917377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxb6l\" (UniqueName: \"kubernetes.io/projected/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-kube-api-access-lxb6l\") pod \"maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:48.917509 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.917407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:48.918088 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.918069    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:48.920004 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.919980    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:48.920113 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.920097    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:48.924369 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.924338    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:48.927218 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:48.927199    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxb6l\" (UniqueName: \"kubernetes.io/projected/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-kube-api-access-lxb6l\") pod \"maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:49.024587 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:49.024519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:49.302975 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:49.302945    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"]
Sep 10 20:20:49.303923 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:20:49.303885    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a564b4_fd2d_43e3_b2ec_3da1c4e53ced.slice/crio-f06e81f5ff0c3f64c929c8cfa7366521959f7a6b0909b9df3207571c49e9f484 WatchSource:0}: Error finding container f06e81f5ff0c3f64c929c8cfa7366521959f7a6b0909b9df3207571c49e9f484: Status 404 returned error can't find the container with id f06e81f5ff0c3f64c929c8cfa7366521959f7a6b0909b9df3207571c49e9f484
Sep 10 20:20:49.333422 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:49.333389    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:49.433701 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:49.433667    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:20:49.952354 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:49.952316    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b" event={"ID":"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced","Type":"ContainerStarted","Data":"f06e81f5ff0c3f64c929c8cfa7366521959f7a6b0909b9df3207571c49e9f484"}
Sep 10 20:20:50.962614 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:50.962576    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b" event={"ID":"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced","Type":"ContainerStarted","Data":"8492a3dc402f2ae1a3e3f1c5323006cdc779bb23f3175490b49e5bcbc5586a29"}
Sep 10 20:20:50.963013 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:50.962643    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:20:50.982026 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:50.981973    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b" podStartSLOduration=2.06500986 podStartE2EDuration="2.981958826s" totalImagesPullingTime="916.948966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:20:50.980322461 +0000 UTC m=+1592.030850451" watchObservedRunningTime="2026-09-10 20:20:50.981958826 +0000 UTC m=+1592.032486814"
Sep 10 20:20:56.974268 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:20:56.974233    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:21:12.432669 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:12.432582    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:19.251035 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.251006    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"]
Sep 10 20:21:19.251516 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.251219    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b" podUID="a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced" containerName="maas-api" containerID="cri-o://8492a3dc402f2ae1a3e3f1c5323006cdc779bb23f3175490b49e5bcbc5586a29" gracePeriod=30
Sep 10 20:21:19.289330 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.289294    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:19.388723 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.388683    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:19.486857 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.486825    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:19.512674 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.512607    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:21:19.557719 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.557669    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:19.614568 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.614513    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-maas-api-tls\") pod \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") "
Sep 10 20:21:19.614747 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.614599    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxb6l\" (UniqueName: \"kubernetes.io/projected/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-kube-api-access-lxb6l\") pod \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") "
Sep 10 20:21:19.614747 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.614622    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-metrics-certs\") pod \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") "
Sep 10 20:21:19.614747 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.614690    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-openshift-service-ca\") pod \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\" (UID: \"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced\") "
Sep 10 20:21:19.615138 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.615109    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced" (UID: "a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:21:19.616779 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.616752    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced" (UID: "a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:21:19.616887 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.616867    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced" (UID: "a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:21:19.616887 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.616873    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-kube-api-access-lxb6l" (OuterVolumeSpecName: "kube-api-access-lxb6l") pod "a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced" (UID: "a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced"). InnerVolumeSpecName "kube-api-access-lxb6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:21:19.716362 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.716323    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:21:19.716362 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.716355    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:21:19.716362 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.716368    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxb6l\" (UniqueName: \"kubernetes.io/projected/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-kube-api-access-lxb6l\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:21:19.716646 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.716380    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:21:19.860121 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:19.860086    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:20.118018 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:20.117925    2561 generic.go:356] "Generic (PLEG): container finished" podID="a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced" containerID="8492a3dc402f2ae1a3e3f1c5323006cdc779bb23f3175490b49e5bcbc5586a29" exitCode=0
Sep 10 20:21:20.118018 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:20.117971    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"
Sep 10 20:21:20.118227 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:20.118013    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b" event={"ID":"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced","Type":"ContainerDied","Data":"8492a3dc402f2ae1a3e3f1c5323006cdc779bb23f3175490b49e5bcbc5586a29"}
Sep 10 20:21:20.118227 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:20.118063    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b" event={"ID":"a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced","Type":"ContainerDied","Data":"f06e81f5ff0c3f64c929c8cfa7366521959f7a6b0909b9df3207571c49e9f484"}
Sep 10 20:21:20.118227 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:20.118086    2561 scope.go:117] "RemoveContainer" containerID="8492a3dc402f2ae1a3e3f1c5323006cdc779bb23f3175490b49e5bcbc5586a29"
Sep 10 20:21:20.128352 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:20.128333    2561 scope.go:117] "RemoveContainer" containerID="8492a3dc402f2ae1a3e3f1c5323006cdc779bb23f3175490b49e5bcbc5586a29"
Sep 10 20:21:20.128670 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:21:20.128650    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8492a3dc402f2ae1a3e3f1c5323006cdc779bb23f3175490b49e5bcbc5586a29\": container with ID starting with 8492a3dc402f2ae1a3e3f1c5323006cdc779bb23f3175490b49e5bcbc5586a29 not found: ID does not exist" containerID="8492a3dc402f2ae1a3e3f1c5323006cdc779bb23f3175490b49e5bcbc5586a29"
Sep 10 20:21:20.128728 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:20.128679    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8492a3dc402f2ae1a3e3f1c5323006cdc779bb23f3175490b49e5bcbc5586a29"} err="failed to get container status \"8492a3dc402f2ae1a3e3f1c5323006cdc779bb23f3175490b49e5bcbc5586a29\": rpc error: code = NotFound desc = could not find container \"8492a3dc402f2ae1a3e3f1c5323006cdc779bb23f3175490b49e5bcbc5586a29\": container with ID starting with 8492a3dc402f2ae1a3e3f1c5323006cdc779bb23f3175490b49e5bcbc5586a29 not found: ID does not exist"
Sep 10 20:21:20.140899 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:20.140871    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"]
Sep 10 20:21:20.145371 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:20.145343    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fe603d-7c45b7fcd6-pfg9b"]
Sep 10 20:21:21.404728 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:21.404685    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced" path="/var/lib/kubelet/pods/a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced/volumes"
Sep 10 20:21:25.529440 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:25.529404    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:25.629138 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:25.629096    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:25.804211 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:25.804127    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:29.561290 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:29.561254    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:44.116408 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:44.116363    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:44.938834 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:44.938802    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"]
Sep 10 20:21:44.939059 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:44.939034    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6" podUID="0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1" containerName="maas-api" containerID="cri-o://a93395d8faa80e8dfc230ad95d52777dc50943d8740f71dad89abb9c0f98776d" gracePeriod=30
Sep 10 20:21:44.972784 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:44.972751    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:45.071917 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.071869    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:45.164201 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.164168    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:45.190915 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.190849    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:21:45.222277 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.222245    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:45.255958 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.255929    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-metrics-certs\") pod \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") "
Sep 10 20:21:45.256166 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.255991    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxgfq\" (UniqueName: \"kubernetes.io/projected/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-kube-api-access-sxgfq\") pod \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") "
Sep 10 20:21:45.256166 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.256030    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-maas-api-tls\") pod \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") "
Sep 10 20:21:45.256166 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.256074    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-openshift-service-ca\") pod \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\" (UID: \"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1\") "
Sep 10 20:21:45.256559 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.256505    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1" (UID: "0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:21:45.258293 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.258266    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1" (UID: "0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:21:45.258398 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.258303    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-kube-api-access-sxgfq" (OuterVolumeSpecName: "kube-api-access-sxgfq") pod "0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1" (UID: "0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1"). InnerVolumeSpecName "kube-api-access-sxgfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:21:45.258608 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.258584    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1" (UID: "0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:21:45.260582 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.260560    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6" event={"ID":"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1","Type":"ContainerDied","Data":"a93395d8faa80e8dfc230ad95d52777dc50943d8740f71dad89abb9c0f98776d"}
Sep 10 20:21:45.260582 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.260568    2561 generic.go:356] "Generic (PLEG): container finished" podID="0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1" containerID="a93395d8faa80e8dfc230ad95d52777dc50943d8740f71dad89abb9c0f98776d" exitCode=0
Sep 10 20:21:45.260716 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.260604    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"
Sep 10 20:21:45.260716 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.260627    2561 scope.go:117] "RemoveContainer" containerID="a93395d8faa80e8dfc230ad95d52777dc50943d8740f71dad89abb9c0f98776d"
Sep 10 20:21:45.260802 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.260617    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6" event={"ID":"0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1","Type":"ContainerDied","Data":"54b562554de1ee5427da425e36e23eef8e2017e509f9f9811716646e78338efc"}
Sep 10 20:21:45.277203 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.277174    2561 scope.go:117] "RemoveContainer" containerID="a93395d8faa80e8dfc230ad95d52777dc50943d8740f71dad89abb9c0f98776d"
Sep 10 20:21:45.277592 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:21:45.277569    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93395d8faa80e8dfc230ad95d52777dc50943d8740f71dad89abb9c0f98776d\": container with ID starting with a93395d8faa80e8dfc230ad95d52777dc50943d8740f71dad89abb9c0f98776d not found: ID does not exist" containerID="a93395d8faa80e8dfc230ad95d52777dc50943d8740f71dad89abb9c0f98776d"
Sep 10 20:21:45.277671 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.277600    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93395d8faa80e8dfc230ad95d52777dc50943d8740f71dad89abb9c0f98776d"} err="failed to get container status \"a93395d8faa80e8dfc230ad95d52777dc50943d8740f71dad89abb9c0f98776d\": rpc error: code = NotFound desc = could not find container \"a93395d8faa80e8dfc230ad95d52777dc50943d8740f71dad89abb9c0f98776d\": container with ID starting with a93395d8faa80e8dfc230ad95d52777dc50943d8740f71dad89abb9c0f98776d not found: ID does not exist"
Sep 10 20:21:45.287845 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.287820    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"]
Sep 10 20:21:45.291062 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.291039    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-316fde-7bcb6f98b4-8dmk6"]
Sep 10 20:21:45.357169 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.357129    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:21:45.357169 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.357161    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:21:45.357169 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.357176    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:21:45.357412 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.357188    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxgfq\" (UniqueName: \"kubernetes.io/projected/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1-kube-api-access-sxgfq\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:21:45.404299 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.404262    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1" path="/var/lib/kubelet/pods/0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1/volumes"
Sep 10 20:21:45.548896 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:45.548865    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:21:50.528587 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:50.528546    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"]
Sep 10 20:21:50.623397 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:21:50.623359    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:05.549675 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.549633    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc" podUID="eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc" containerName="main" containerID="cri-o://860c2f6835e60627f5c410d813d43a950a9e378d94f0abdfc717e7abd7cecce7" gracePeriod=45
Sep 10 20:22:05.800751 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.800689    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:22:05.849629 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.849594    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-home\") pod \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") "
Sep 10 20:22:05.849804 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.849636    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-model-cache\") pod \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") "
Sep 10 20:22:05.849804 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.849668    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6df6j\" (UniqueName: \"kubernetes.io/projected/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-kube-api-access-6df6j\") pod \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") "
Sep 10 20:22:05.849804 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.849716    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-dshm\") pod \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") "
Sep 10 20:22:05.849804 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.849745    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-tls-certs\") pod \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") "
Sep 10 20:22:05.850041 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.849839    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-tmp-dir\") pod \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\" (UID: \"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc\") "
Sep 10 20:22:05.850041 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.849896    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-home" (OuterVolumeSpecName: "home") pod "eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc" (UID: "eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:22:05.850041 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.849970    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc" (UID: "eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:22:05.850202 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.850098    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc" (UID: "eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:22:05.850264 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.850217    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:22:05.850264 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.850233    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:22:05.850264 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.850248    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:22:05.852369 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.852346    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-dshm" (OuterVolumeSpecName: "dshm") pod "eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc" (UID: "eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:22:05.852587 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.852567    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc" (UID: "eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:22:05.852587 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.852580    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-kube-api-access-6df6j" (OuterVolumeSpecName: "kube-api-access-6df6j") pod "eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc" (UID: "eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc"). InnerVolumeSpecName "kube-api-access-6df6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:22:05.951191 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.951150    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6df6j\" (UniqueName: \"kubernetes.io/projected/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-kube-api-access-6df6j\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:22:05.951191 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.951185    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:22:05.951191 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:05.951194    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:22:06.376415 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:06.376377    2561 generic.go:356] "Generic (PLEG): container finished" podID="eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc" containerID="860c2f6835e60627f5c410d813d43a950a9e378d94f0abdfc717e7abd7cecce7" exitCode=0
Sep 10 20:22:06.376587 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:06.376421    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"
Sep 10 20:22:06.376587 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:06.376460    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc" event={"ID":"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc","Type":"ContainerDied","Data":"860c2f6835e60627f5c410d813d43a950a9e378d94f0abdfc717e7abd7cecce7"}
Sep 10 20:22:06.376587 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:06.376505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc" event={"ID":"eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc","Type":"ContainerDied","Data":"862e2670846e1235a700221f9bffd61a5f0e45753bdfd7740eeb262e4085435d"}
Sep 10 20:22:06.376587 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:06.376523    2561 scope.go:117] "RemoveContainer" containerID="860c2f6835e60627f5c410d813d43a950a9e378d94f0abdfc717e7abd7cecce7"
Sep 10 20:22:06.386442 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:06.386426    2561 scope.go:117] "RemoveContainer" containerID="860c2f6835e60627f5c410d813d43a950a9e378d94f0abdfc717e7abd7cecce7"
Sep 10 20:22:06.386750 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:22:06.386733    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"860c2f6835e60627f5c410d813d43a950a9e378d94f0abdfc717e7abd7cecce7\": container with ID starting with 860c2f6835e60627f5c410d813d43a950a9e378d94f0abdfc717e7abd7cecce7 not found: ID does not exist" containerID="860c2f6835e60627f5c410d813d43a950a9e378d94f0abdfc717e7abd7cecce7"
Sep 10 20:22:06.386816 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:06.386760    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860c2f6835e60627f5c410d813d43a950a9e378d94f0abdfc717e7abd7cecce7"} err="failed to get container status \"860c2f6835e60627f5c410d813d43a950a9e378d94f0abdfc717e7abd7cecce7\": rpc error: code = NotFound desc = could not find container \"860c2f6835e60627f5c410d813d43a950a9e378d94f0abdfc717e7abd7cecce7\": container with ID starting with 860c2f6835e60627f5c410d813d43a950a9e378d94f0abdfc717e7abd7cecce7 not found: ID does not exist"
Sep 10 20:22:06.398137 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:06.398111    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"]
Sep 10 20:22:06.402020 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:06.401995    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-316fde/ipp-route-316fde-kserve-6c974887-njhpc"]
Sep 10 20:22:07.403288 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:07.403252    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc" path="/var/lib/kubelet/pods/eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc/volumes"
Sep 10 20:22:13.618230 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:13.618192    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:26.705971 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:26.705936    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:27.542007 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:27.541975    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:33.155819 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:33.155782    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:33.220992 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:33.220948    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:33.429733 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:33.429623    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:33.897036 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:33.896995    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:44.184114 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.184020    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"]
Sep 10 20:22:44.184604 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.184278    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d" podUID="c7b3683d-e5f9-4a37-bf25-f2693640d3a0" containerName="maas-api" containerID="cri-o://aae05741e5081926873c4cbd613d605485346962f00af1ac776072120ef2015b" gracePeriod=30
Sep 10 20:22:44.258661 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.258623    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:44.397863 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.397827    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:44.429137 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.429115    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:22:44.497311 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.497228    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:44.582807 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.582770    2561 generic.go:356] "Generic (PLEG): container finished" podID="c7b3683d-e5f9-4a37-bf25-f2693640d3a0" containerID="aae05741e5081926873c4cbd613d605485346962f00af1ac776072120ef2015b" exitCode=0
Sep 10 20:22:44.582991 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.582819    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"
Sep 10 20:22:44.582991 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.582863    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d" event={"ID":"c7b3683d-e5f9-4a37-bf25-f2693640d3a0","Type":"ContainerDied","Data":"aae05741e5081926873c4cbd613d605485346962f00af1ac776072120ef2015b"}
Sep 10 20:22:44.582991 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.582910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d" event={"ID":"c7b3683d-e5f9-4a37-bf25-f2693640d3a0","Type":"ContainerDied","Data":"eba3aaf1db6bb9f0afba765236288cb6414c996823c0766f8fd3a543b10fb4ff"}
Sep 10 20:22:44.582991 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.582931    2561 scope.go:117] "RemoveContainer" containerID="aae05741e5081926873c4cbd613d605485346962f00af1ac776072120ef2015b"
Sep 10 20:22:44.591737 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.591722    2561 scope.go:117] "RemoveContainer" containerID="aae05741e5081926873c4cbd613d605485346962f00af1ac776072120ef2015b"
Sep 10 20:22:44.592025 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:22:44.592009    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae05741e5081926873c4cbd613d605485346962f00af1ac776072120ef2015b\": container with ID starting with aae05741e5081926873c4cbd613d605485346962f00af1ac776072120ef2015b not found: ID does not exist" containerID="aae05741e5081926873c4cbd613d605485346962f00af1ac776072120ef2015b"
Sep 10 20:22:44.592070 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.592034    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae05741e5081926873c4cbd613d605485346962f00af1ac776072120ef2015b"} err="failed to get container status \"aae05741e5081926873c4cbd613d605485346962f00af1ac776072120ef2015b\": rpc error: code = NotFound desc = could not find container \"aae05741e5081926873c4cbd613d605485346962f00af1ac776072120ef2015b\": container with ID starting with aae05741e5081926873c4cbd613d605485346962f00af1ac776072120ef2015b not found: ID does not exist"
Sep 10 20:22:44.603500 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.603481    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:44.608577 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.608558    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-maas-api-tls\") pod \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") "
Sep 10 20:22:44.608679 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.608625    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-openshift-service-ca\") pod \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") "
Sep 10 20:22:44.608679 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.608667    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-metrics-certs\") pod \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") "
Sep 10 20:22:44.608795 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.608729    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wgxr\" (UniqueName: \"kubernetes.io/projected/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-kube-api-access-8wgxr\") pod \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\" (UID: \"c7b3683d-e5f9-4a37-bf25-f2693640d3a0\") "
Sep 10 20:22:44.609018 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.608995    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7b3683d-e5f9-4a37-bf25-f2693640d3a0" (UID: "c7b3683d-e5f9-4a37-bf25-f2693640d3a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:22:44.610867 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.610842    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7b3683d-e5f9-4a37-bf25-f2693640d3a0" (UID: "c7b3683d-e5f9-4a37-bf25-f2693640d3a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:22:44.610954 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.610863    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-kube-api-access-8wgxr" (OuterVolumeSpecName: "kube-api-access-8wgxr") pod "c7b3683d-e5f9-4a37-bf25-f2693640d3a0" (UID: "c7b3683d-e5f9-4a37-bf25-f2693640d3a0"). InnerVolumeSpecName "kube-api-access-8wgxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:22:44.610954 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.610930    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c7b3683d-e5f9-4a37-bf25-f2693640d3a0" (UID: "c7b3683d-e5f9-4a37-bf25-f2693640d3a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:22:44.709875 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.709822    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:22:44.709875 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.709865    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:22:44.709875 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.709875    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wgxr\" (UniqueName: \"kubernetes.io/projected/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-kube-api-access-8wgxr\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:22:44.709875 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.709883    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b3683d-e5f9-4a37-bf25-f2693640d3a0-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:22:44.891158 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.891128    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:44.904140 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.904111    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"]
Sep 10 20:22:44.907825 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:44.907798    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-82457a-664df89fcf-6685d"]
Sep 10 20:22:45.405016 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:45.404983    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7b3683d-e5f9-4a37-bf25-f2693640d3a0" path="/var/lib/kubelet/pods/c7b3683d-e5f9-4a37-bf25-f2693640d3a0/volumes"
Sep 10 20:22:50.704118 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:50.704082    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:50.804198 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:50.804164    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:50.973835 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:50.973710    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:22:51.525950 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:22:51.525921    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:01.817249 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:01.817214    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"]
Sep 10 20:23:01.817786 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:01.817447    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps" podUID="59419042-05a6-477e-ae6b-f6eab5eda4dc" containerName="maas-api" containerID="cri-o://90bcea8240a080a34a02431f936b89e7caa947a63a4bf9676a4e3a96f6d7a72b" gracePeriod=30
Sep 10 20:23:01.867584 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:01.867546    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:01.951593 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:01.951492    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:02.051372 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.051342    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:02.070576 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.070512    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:23:02.150964 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.150927    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59419042-05a6-477e-ae6b-f6eab5eda4dc-openshift-service-ca\") pod \"59419042-05a6-477e-ae6b-f6eab5eda4dc\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") "
Sep 10 20:23:02.151171 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.150989    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59419042-05a6-477e-ae6b-f6eab5eda4dc-maas-api-tls\") pod \"59419042-05a6-477e-ae6b-f6eab5eda4dc\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") "
Sep 10 20:23:02.151171 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.151015    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59419042-05a6-477e-ae6b-f6eab5eda4dc-metrics-certs\") pod \"59419042-05a6-477e-ae6b-f6eab5eda4dc\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") "
Sep 10 20:23:02.151171 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.151101    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wmp6\" (UniqueName: \"kubernetes.io/projected/59419042-05a6-477e-ae6b-f6eab5eda4dc-kube-api-access-2wmp6\") pod \"59419042-05a6-477e-ae6b-f6eab5eda4dc\" (UID: \"59419042-05a6-477e-ae6b-f6eab5eda4dc\") "
Sep 10 20:23:02.151393 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.151368    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59419042-05a6-477e-ae6b-f6eab5eda4dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "59419042-05a6-477e-ae6b-f6eab5eda4dc" (UID: "59419042-05a6-477e-ae6b-f6eab5eda4dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:23:02.152056 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.152027    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:02.153430 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.153405    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59419042-05a6-477e-ae6b-f6eab5eda4dc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "59419042-05a6-477e-ae6b-f6eab5eda4dc" (UID: "59419042-05a6-477e-ae6b-f6eab5eda4dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:23:02.153515 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.153420    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59419042-05a6-477e-ae6b-f6eab5eda4dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "59419042-05a6-477e-ae6b-f6eab5eda4dc" (UID: "59419042-05a6-477e-ae6b-f6eab5eda4dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:23:02.153515 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.153425    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59419042-05a6-477e-ae6b-f6eab5eda4dc-kube-api-access-2wmp6" (OuterVolumeSpecName: "kube-api-access-2wmp6") pod "59419042-05a6-477e-ae6b-f6eab5eda4dc" (UID: "59419042-05a6-477e-ae6b-f6eab5eda4dc"). InnerVolumeSpecName "kube-api-access-2wmp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:23:02.252700 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.252667    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wmp6\" (UniqueName: \"kubernetes.io/projected/59419042-05a6-477e-ae6b-f6eab5eda4dc-kube-api-access-2wmp6\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:23:02.252700 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.252698    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59419042-05a6-477e-ae6b-f6eab5eda4dc-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:23:02.252907 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.252711    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59419042-05a6-477e-ae6b-f6eab5eda4dc-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:23:02.252907 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.252724    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59419042-05a6-477e-ae6b-f6eab5eda4dc-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:23:02.457238 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.457148    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:02.677003 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.676964    2561 generic.go:356] "Generic (PLEG): container finished" podID="59419042-05a6-477e-ae6b-f6eab5eda4dc" containerID="90bcea8240a080a34a02431f936b89e7caa947a63a4bf9676a4e3a96f6d7a72b" exitCode=0
Sep 10 20:23:02.677173 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.677013    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"
Sep 10 20:23:02.677173 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.677058    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps" event={"ID":"59419042-05a6-477e-ae6b-f6eab5eda4dc","Type":"ContainerDied","Data":"90bcea8240a080a34a02431f936b89e7caa947a63a4bf9676a4e3a96f6d7a72b"}
Sep 10 20:23:02.677173 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.677108    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps" event={"ID":"59419042-05a6-477e-ae6b-f6eab5eda4dc","Type":"ContainerDied","Data":"328a0a4e7af94868d8dd45dfb0561b8b285a37b4336de86389ede2239e28a9b7"}
Sep 10 20:23:02.677173 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.677134    2561 scope.go:117] "RemoveContainer" containerID="90bcea8240a080a34a02431f936b89e7caa947a63a4bf9676a4e3a96f6d7a72b"
Sep 10 20:23:02.688109 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.688090    2561 scope.go:117] "RemoveContainer" containerID="90bcea8240a080a34a02431f936b89e7caa947a63a4bf9676a4e3a96f6d7a72b"
Sep 10 20:23:02.688413 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:23:02.688393    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90bcea8240a080a34a02431f936b89e7caa947a63a4bf9676a4e3a96f6d7a72b\": container with ID starting with 90bcea8240a080a34a02431f936b89e7caa947a63a4bf9676a4e3a96f6d7a72b not found: ID does not exist" containerID="90bcea8240a080a34a02431f936b89e7caa947a63a4bf9676a4e3a96f6d7a72b"
Sep 10 20:23:02.688469 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.688422    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90bcea8240a080a34a02431f936b89e7caa947a63a4bf9676a4e3a96f6d7a72b"} err="failed to get container status \"90bcea8240a080a34a02431f936b89e7caa947a63a4bf9676a4e3a96f6d7a72b\": rpc error: code = NotFound desc = could not find container \"90bcea8240a080a34a02431f936b89e7caa947a63a4bf9676a4e3a96f6d7a72b\": container with ID starting with 90bcea8240a080a34a02431f936b89e7caa947a63a4bf9676a4e3a96f6d7a72b not found: ID does not exist"
Sep 10 20:23:02.699029 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.698995    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"]
Sep 10 20:23:02.704856 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:02.704832    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-591d7c-6496b95d95-dgnps"]
Sep 10 20:23:03.404307 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:03.404273    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59419042-05a6-477e-ae6b-f6eab5eda4dc" path="/var/lib/kubelet/pods/59419042-05a6-477e-ae6b-f6eab5eda4dc/volumes"
Sep 10 20:23:08.488390 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:08.488346    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:08.557620 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:08.557588    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:08.788691 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:08.788650    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:09.219893 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:09.219811    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:19.535833 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.535804    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"]
Sep 10 20:23:19.536289 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.536061    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb" podUID="21e127b8-bdca-4dbc-848e-afb20ad8fb00" containerName="maas-api" containerID="cri-o://64220f7c669e155b14799de79f985703dc16ac19652220a5437d9030a9b05fae" gracePeriod=30
Sep 10 20:23:19.568056 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.568025    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:19.688270 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.688241    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:19.766484 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.766449    2561 generic.go:356] "Generic (PLEG): container finished" podID="21e127b8-bdca-4dbc-848e-afb20ad8fb00" containerID="64220f7c669e155b14799de79f985703dc16ac19652220a5437d9030a9b05fae" exitCode=0
Sep 10 20:23:19.766626 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.766550    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb" event={"ID":"21e127b8-bdca-4dbc-848e-afb20ad8fb00","Type":"ContainerDied","Data":"64220f7c669e155b14799de79f985703dc16ac19652220a5437d9030a9b05fae"}
Sep 10 20:23:19.766626 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.766607    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb" event={"ID":"21e127b8-bdca-4dbc-848e-afb20ad8fb00","Type":"ContainerDied","Data":"d2162bd29cf8cd1be17acab775a3a25def7d7342a832c73c1a24ddc36160c264"}
Sep 10 20:23:19.766626 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.766625    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2162bd29cf8cd1be17acab775a3a25def7d7342a832c73c1a24ddc36160c264"
Sep 10 20:23:19.782139 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.782111    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:23:19.794482 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.794387    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:19.876446 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.876382    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:19.914598 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.914574    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e127b8-bdca-4dbc-848e-afb20ad8fb00-maas-api-tls\") pod \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") "
Sep 10 20:23:19.914786 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.914613    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsxbt\" (UniqueName: \"kubernetes.io/projected/21e127b8-bdca-4dbc-848e-afb20ad8fb00-kube-api-access-zsxbt\") pod \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") "
Sep 10 20:23:19.914786 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.914681    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e127b8-bdca-4dbc-848e-afb20ad8fb00-metrics-certs\") pod \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") "
Sep 10 20:23:19.914786 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.914746    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e127b8-bdca-4dbc-848e-afb20ad8fb00-openshift-service-ca\") pod \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\" (UID: \"21e127b8-bdca-4dbc-848e-afb20ad8fb00\") "
Sep 10 20:23:19.915207 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.915175    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e127b8-bdca-4dbc-848e-afb20ad8fb00-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "21e127b8-bdca-4dbc-848e-afb20ad8fb00" (UID: "21e127b8-bdca-4dbc-848e-afb20ad8fb00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:23:19.917244 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.917221    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e127b8-bdca-4dbc-848e-afb20ad8fb00-kube-api-access-zsxbt" (OuterVolumeSpecName: "kube-api-access-zsxbt") pod "21e127b8-bdca-4dbc-848e-afb20ad8fb00" (UID: "21e127b8-bdca-4dbc-848e-afb20ad8fb00"). InnerVolumeSpecName "kube-api-access-zsxbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:23:19.917575 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.917546    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e127b8-bdca-4dbc-848e-afb20ad8fb00-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "21e127b8-bdca-4dbc-848e-afb20ad8fb00" (UID: "21e127b8-bdca-4dbc-848e-afb20ad8fb00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:23:19.917575 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.917557    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e127b8-bdca-4dbc-848e-afb20ad8fb00-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "21e127b8-bdca-4dbc-848e-afb20ad8fb00" (UID: "21e127b8-bdca-4dbc-848e-afb20ad8fb00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:23:19.933049 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:19.933026    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:20.015601 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:20.015566    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsxbt\" (UniqueName: \"kubernetes.io/projected/21e127b8-bdca-4dbc-848e-afb20ad8fb00-kube-api-access-zsxbt\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:23:20.015601 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:20.015600    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e127b8-bdca-4dbc-848e-afb20ad8fb00-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:23:20.015601 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:20.015609    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e127b8-bdca-4dbc-848e-afb20ad8fb00-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:23:20.015878 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:20.015627    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e127b8-bdca-4dbc-848e-afb20ad8fb00-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:23:20.260997 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:20.260914    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:20.772078 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:20.772036    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"
Sep 10 20:23:20.792495 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:20.792463    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"]
Sep 10 20:23:20.796344 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:20.796306    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6550b4-5d5dcd89df-844qb"]
Sep 10 20:23:21.404505 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:21.404475    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21e127b8-bdca-4dbc-848e-afb20ad8fb00" path="/var/lib/kubelet/pods/21e127b8-bdca-4dbc-848e-afb20ad8fb00/volumes"
Sep 10 20:23:26.294071 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:26.294039    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:26.391346 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:26.391311    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:26.577623 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:26.577508    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:45.173180 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:45.173141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:23:45.229819 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:23:45.229789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:24:19.741692 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:24:19.741659    2561 scope.go:117] "RemoveContainer" containerID="64220f7c669e155b14799de79f985703dc16ac19652220a5437d9030a9b05fae"
Sep 10 20:24:20.919078 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:24:20.918957    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 20:24:20.928481 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:24:20.928455    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 20:24:50.741423 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:24:50.741385    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:24:50.970664 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:24:50.970634    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:24:52.845825 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:24:52.845786    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:27:51.860408 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:51.860376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:27:52.752702 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.752668    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"]
Sep 10 20:27:52.753170 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753142    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21e127b8-bdca-4dbc-848e-afb20ad8fb00" containerName="maas-api"
Sep 10 20:27:52.753170 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753166    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="21e127b8-bdca-4dbc-848e-afb20ad8fb00" containerName="maas-api"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753181    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced" containerName="maas-api"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753187    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced" containerName="maas-api"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753202    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1" containerName="maas-api"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753208    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1" containerName="maas-api"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753217    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b3683d-e5f9-4a37-bf25-f2693640d3a0" containerName="maas-api"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753224    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b3683d-e5f9-4a37-bf25-f2693640d3a0" containerName="maas-api"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753242    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59419042-05a6-477e-ae6b-f6eab5eda4dc" containerName="maas-api"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753249    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59419042-05a6-477e-ae6b-f6eab5eda4dc" containerName="maas-api"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753265    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc" containerName="main"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753272    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc" containerName="main"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753351    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b3683d-e5f9-4a37-bf25-f2693640d3a0" containerName="maas-api"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753360    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb97d1a3-8b3d-4108-b264-9a1b6e0a05cc" containerName="main"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753366    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5a564b4-fd2d-43e3-b2ec-3da1c4e53ced" containerName="maas-api"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753372    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="21e127b8-bdca-4dbc-848e-afb20ad8fb00" containerName="maas-api"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753378    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="59419042-05a6-477e-ae6b-f6eab5eda4dc" containerName="maas-api"
Sep 10 20:27:52.753392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.753385    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fd656c0-0e5a-4937-ac1e-fa0af2d0ddd1" containerName="maas-api"
Sep 10 20:27:52.756506 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.756487    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.759092 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.759071    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-15343da8-kserve-self-signed-certs\""
Sep 10 20:27:52.766352 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.766327    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"]
Sep 10 20:27:52.849115 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.849080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-model-cache\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.849115 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.849122    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-home\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.849330 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.849142    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-dshm\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.849330 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.849217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-tls-certs\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.849330 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.849251    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-tmp-dir\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.849330 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.849320    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8th6l\" (UniqueName: \"kubernetes.io/projected/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-kube-api-access-8th6l\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.950598 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.950558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8th6l\" (UniqueName: \"kubernetes.io/projected/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-kube-api-access-8th6l\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.951008 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.950638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-model-cache\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.951008 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.950679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-home\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.951008 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.950708    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-dshm\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.951008 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.950746    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-tls-certs\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.951008 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.950772    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-tmp-dir\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.951177 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.951129    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-model-cache\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.951216 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.951136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-tmp-dir\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.951216 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.951189    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-home\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.953253 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.953221    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-dshm\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.953497 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.953480    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-tls-certs\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:52.958281 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:52.958258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8th6l\" (UniqueName: \"kubernetes.io/projected/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-kube-api-access-8th6l\") pod \"e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") " pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:53.068190 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:53.068150    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:27:53.184085 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:53.184051    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:27:53.342496 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:53.342467    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"]
Sep 10 20:27:53.344908 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:27:53.344885    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dd5df27_2616_4f3d_8acd_654c3b8c8f54.slice/crio-2596f105747a8260ec9b4d6c47753077390164bca5f978b3a0b9a2c0a16bd639 WatchSource:0}: Error finding container 2596f105747a8260ec9b4d6c47753077390164bca5f978b3a0b9a2c0a16bd639: Status 404 returned error can't find the container with id 2596f105747a8260ec9b4d6c47753077390164bca5f978b3a0b9a2c0a16bd639
Sep 10 20:27:54.201753 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:54.201713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d" event={"ID":"2dd5df27-2616-4f3d-8acd-654c3b8c8f54","Type":"ContainerStarted","Data":"efc4b4e462843e67124f884e12c69dcf10a52d903c5adf8800a75b0a542f0c21"}
Sep 10 20:27:54.202124 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:54.201763    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d" event={"ID":"2dd5df27-2616-4f3d-8acd-654c3b8c8f54","Type":"ContainerStarted","Data":"2596f105747a8260ec9b4d6c47753077390164bca5f978b3a0b9a2c0a16bd639"}
Sep 10 20:27:54.220188 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:27:54.220138    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d" podStartSLOduration=2.220122846 podStartE2EDuration="2.220122846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:27:54.217952284 +0000 UTC m=+2015.268480270" watchObservedRunningTime="2026-09-10 20:27:54.220122846 +0000 UTC m=+2015.270650835"
Sep 10 20:28:03.068549 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:03.068480    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:28:03.068992 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:03.068573    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:28:03.081337 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:03.081304    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:28:03.258910 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:03.258882    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:28:08.546420 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.546380    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"]
Sep 10 20:28:08.550676 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.550654    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.553432 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.553399    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-15343da8-kserve-self-signed-certs\""
Sep 10 20:28:08.558072 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.558046    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"]
Sep 10 20:28:08.583502 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.583471    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:28:08.691670 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.691617    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-tmp-dir\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.691860 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.691689    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbmcb\" (UniqueName: \"kubernetes.io/projected/be00da51-8219-4cae-94b4-717851f75c79-kube-api-access-jbmcb\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.691860 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.691753    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-model-cache\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.691860 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.691788    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-dshm\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.691860 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.691822    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be00da51-8219-4cae-94b4-717851f75c79-tls-certs\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.691860 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.691849    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-home\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.792573 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.792506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-dshm\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.792770 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.792587    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be00da51-8219-4cae-94b4-717851f75c79-tls-certs\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.792770 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.792604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-home\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.792770 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.792664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-tmp-dir\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.792770 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.792700    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbmcb\" (UniqueName: \"kubernetes.io/projected/be00da51-8219-4cae-94b4-717851f75c79-kube-api-access-jbmcb\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.792770 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.792721    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-model-cache\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.793062 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.793017    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-home\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.793136 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.793106    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-model-cache\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.793703 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.793675    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-tmp-dir\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.795042 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.795018    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-dshm\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.795321 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.795302    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be00da51-8219-4cae-94b4-717851f75c79-tls-certs\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.801212 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.801154    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbmcb\" (UniqueName: \"kubernetes.io/projected/be00da51-8219-4cae-94b4-717851f75c79-kube-api-access-jbmcb\") pod \"e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") " pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:08.861418 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:08.861370    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:09.138765 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:09.138739    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"]
Sep 10 20:28:09.140213 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:28:09.140180    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe00da51_8219_4cae_94b4_717851f75c79.slice/crio-ec1d8a507490dbea9e89f6b2b2daa0cd0071afd65f5e70886d5a3dbc2cdc4c25 WatchSource:0}: Error finding container ec1d8a507490dbea9e89f6b2b2daa0cd0071afd65f5e70886d5a3dbc2cdc4c25: Status 404 returned error can't find the container with id ec1d8a507490dbea9e89f6b2b2daa0cd0071afd65f5e70886d5a3dbc2cdc4c25
Sep 10 20:28:09.280013 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:09.279979    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh" event={"ID":"be00da51-8219-4cae-94b4-717851f75c79","Type":"ContainerStarted","Data":"fb4514fd05c488687648ce5efce753540d28890601dc6ac683914b3a479979df"}
Sep 10 20:28:09.280013 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:09.280019    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh" event={"ID":"be00da51-8219-4cae-94b4-717851f75c79","Type":"ContainerStarted","Data":"ec1d8a507490dbea9e89f6b2b2daa0cd0071afd65f5e70886d5a3dbc2cdc4c25"}
Sep 10 20:28:09.300025 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:09.299969    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh" podStartSLOduration=1.29995262 podStartE2EDuration="1.29995262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:28:09.296867086 +0000 UTC m=+2030.347395086" watchObservedRunningTime="2026-09-10 20:28:09.29995262 +0000 UTC m=+2030.350480608"
Sep 10 20:28:11.734704 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:11.734668    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:28:11.836324 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:11.836284    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:28:18.861806 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:18.861766    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:18.862427 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:18.861822    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:18.874270 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:18.874249    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:19.343713 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:19.343681    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:29.796310 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:29.796267    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"]
Sep 10 20:28:29.816792 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:29.816760    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:28:31.864670 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:31.864627    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"]
Sep 10 20:28:31.886600 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:31.886571    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:28:44.817195 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:44.817097    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d" podUID="2dd5df27-2616-4f3d-8acd-654c3b8c8f54" containerName="main" containerID="cri-o://efc4b4e462843e67124f884e12c69dcf10a52d903c5adf8800a75b0a542f0c21" gracePeriod=45
Sep 10 20:28:45.065504 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.065480    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:28:45.120133 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.120044    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-model-cache\") pod \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") "
Sep 10 20:28:45.120133 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.120100    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-tls-certs\") pod \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") "
Sep 10 20:28:45.120133 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.120127    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-home\") pod \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") "
Sep 10 20:28:45.120439 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.120149    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-dshm\") pod \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") "
Sep 10 20:28:45.120439 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.120181    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-tmp-dir\") pod \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") "
Sep 10 20:28:45.120439 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.120198    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8th6l\" (UniqueName: \"kubernetes.io/projected/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-kube-api-access-8th6l\") pod \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\" (UID: \"2dd5df27-2616-4f3d-8acd-654c3b8c8f54\") "
Sep 10 20:28:45.120439 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.120338    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-model-cache" (OuterVolumeSpecName: "model-cache") pod "2dd5df27-2616-4f3d-8acd-654c3b8c8f54" (UID: "2dd5df27-2616-4f3d-8acd-654c3b8c8f54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:28:45.120654 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.120471    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-home" (OuterVolumeSpecName: "home") pod "2dd5df27-2616-4f3d-8acd-654c3b8c8f54" (UID: "2dd5df27-2616-4f3d-8acd-654c3b8c8f54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:28:45.120654 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.120490    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2dd5df27-2616-4f3d-8acd-654c3b8c8f54" (UID: "2dd5df27-2616-4f3d-8acd-654c3b8c8f54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:28:45.120654 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.120587    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:28:45.120654 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.120599    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:28:45.120654 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.120607    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:28:45.122509 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.122486    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-dshm" (OuterVolumeSpecName: "dshm") pod "2dd5df27-2616-4f3d-8acd-654c3b8c8f54" (UID: "2dd5df27-2616-4f3d-8acd-654c3b8c8f54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:28:45.122642 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.122505    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-kube-api-access-8th6l" (OuterVolumeSpecName: "kube-api-access-8th6l") pod "2dd5df27-2616-4f3d-8acd-654c3b8c8f54" (UID: "2dd5df27-2616-4f3d-8acd-654c3b8c8f54"). InnerVolumeSpecName "kube-api-access-8th6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:28:45.122642 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.122553    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2dd5df27-2616-4f3d-8acd-654c3b8c8f54" (UID: "2dd5df27-2616-4f3d-8acd-654c3b8c8f54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:28:45.221296 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.221258    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:28:45.221296 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.221288    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:28:45.221296 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.221296    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8th6l\" (UniqueName: \"kubernetes.io/projected/2dd5df27-2616-4f3d-8acd-654c3b8c8f54-kube-api-access-8th6l\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:28:45.470264 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.470224    2561 generic.go:356] "Generic (PLEG): container finished" podID="2dd5df27-2616-4f3d-8acd-654c3b8c8f54" containerID="efc4b4e462843e67124f884e12c69dcf10a52d903c5adf8800a75b0a542f0c21" exitCode=0
Sep 10 20:28:45.470435 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.470272    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"
Sep 10 20:28:45.470435 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.470307    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d" event={"ID":"2dd5df27-2616-4f3d-8acd-654c3b8c8f54","Type":"ContainerDied","Data":"efc4b4e462843e67124f884e12c69dcf10a52d903c5adf8800a75b0a542f0c21"}
Sep 10 20:28:45.470435 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.470350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d" event={"ID":"2dd5df27-2616-4f3d-8acd-654c3b8c8f54","Type":"ContainerDied","Data":"2596f105747a8260ec9b4d6c47753077390164bca5f978b3a0b9a2c0a16bd639"}
Sep 10 20:28:45.470435 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.470368    2561 scope.go:117] "RemoveContainer" containerID="efc4b4e462843e67124f884e12c69dcf10a52d903c5adf8800a75b0a542f0c21"
Sep 10 20:28:45.479611 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.479591    2561 scope.go:117] "RemoveContainer" containerID="efc4b4e462843e67124f884e12c69dcf10a52d903c5adf8800a75b0a542f0c21"
Sep 10 20:28:45.479887 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:28:45.479865    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc4b4e462843e67124f884e12c69dcf10a52d903c5adf8800a75b0a542f0c21\": container with ID starting with efc4b4e462843e67124f884e12c69dcf10a52d903c5adf8800a75b0a542f0c21 not found: ID does not exist" containerID="efc4b4e462843e67124f884e12c69dcf10a52d903c5adf8800a75b0a542f0c21"
Sep 10 20:28:45.479946 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.479901    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc4b4e462843e67124f884e12c69dcf10a52d903c5adf8800a75b0a542f0c21"} err="failed to get container status \"efc4b4e462843e67124f884e12c69dcf10a52d903c5adf8800a75b0a542f0c21\": rpc error: code = NotFound desc = could not find container \"efc4b4e462843e67124f884e12c69dcf10a52d903c5adf8800a75b0a542f0c21\": container with ID starting with efc4b4e462843e67124f884e12c69dcf10a52d903c5adf8800a75b0a542f0c21 not found: ID does not exist"
Sep 10 20:28:45.488264 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.488241    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"]
Sep 10 20:28:45.497071 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:45.497042    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-15343da8-kserve-7d8998f947-hz59d"]
Sep 10 20:28:46.777266 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:46.777230    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:28:46.885511 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:46.885469    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh" podUID="be00da51-8219-4cae-94b4-717851f75c79" containerName="main" containerID="cri-o://fb4514fd05c488687648ce5efce753540d28890601dc6ac683914b3a479979df" gracePeriod=45
Sep 10 20:28:47.137729 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.137702    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:47.238760 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.238730    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-model-cache\") pod \"be00da51-8219-4cae-94b4-717851f75c79\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") "
Sep 10 20:28:47.238924 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.238802    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be00da51-8219-4cae-94b4-717851f75c79-tls-certs\") pod \"be00da51-8219-4cae-94b4-717851f75c79\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") "
Sep 10 20:28:47.238924 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.238847    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-tmp-dir\") pod \"be00da51-8219-4cae-94b4-717851f75c79\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") "
Sep 10 20:28:47.238924 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.238873    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbmcb\" (UniqueName: \"kubernetes.io/projected/be00da51-8219-4cae-94b4-717851f75c79-kube-api-access-jbmcb\") pod \"be00da51-8219-4cae-94b4-717851f75c79\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") "
Sep 10 20:28:47.238924 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.238903    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-dshm\") pod \"be00da51-8219-4cae-94b4-717851f75c79\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") "
Sep 10 20:28:47.239109 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.238937    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-home\") pod \"be00da51-8219-4cae-94b4-717851f75c79\" (UID: \"be00da51-8219-4cae-94b4-717851f75c79\") "
Sep 10 20:28:47.239109 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.239040    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-model-cache" (OuterVolumeSpecName: "model-cache") pod "be00da51-8219-4cae-94b4-717851f75c79" (UID: "be00da51-8219-4cae-94b4-717851f75c79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:28:47.239213 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.239159    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be00da51-8219-4cae-94b4-717851f75c79" (UID: "be00da51-8219-4cae-94b4-717851f75c79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:28:47.239296 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.239267    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:28:47.239296 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.239292    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:28:47.239394 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.239332    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-home" (OuterVolumeSpecName: "home") pod "be00da51-8219-4cae-94b4-717851f75c79" (UID: "be00da51-8219-4cae-94b4-717851f75c79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:28:47.241237 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.241212    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-dshm" (OuterVolumeSpecName: "dshm") pod "be00da51-8219-4cae-94b4-717851f75c79" (UID: "be00da51-8219-4cae-94b4-717851f75c79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:28:47.241237 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.241221    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be00da51-8219-4cae-94b4-717851f75c79-kube-api-access-jbmcb" (OuterVolumeSpecName: "kube-api-access-jbmcb") pod "be00da51-8219-4cae-94b4-717851f75c79" (UID: "be00da51-8219-4cae-94b4-717851f75c79"). InnerVolumeSpecName "kube-api-access-jbmcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:28:47.241387 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.241239    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be00da51-8219-4cae-94b4-717851f75c79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be00da51-8219-4cae-94b4-717851f75c79" (UID: "be00da51-8219-4cae-94b4-717851f75c79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:28:47.339959 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.339868    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbmcb\" (UniqueName: \"kubernetes.io/projected/be00da51-8219-4cae-94b4-717851f75c79-kube-api-access-jbmcb\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:28:47.339959 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.339900    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:28:47.339959 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.339911    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be00da51-8219-4cae-94b4-717851f75c79-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:28:47.339959 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.339919    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be00da51-8219-4cae-94b4-717851f75c79-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:28:47.404343 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.404310    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd5df27-2616-4f3d-8acd-654c3b8c8f54" path="/var/lib/kubelet/pods/2dd5df27-2616-4f3d-8acd-654c3b8c8f54/volumes"
Sep 10 20:28:47.487484 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.487448    2561 generic.go:356] "Generic (PLEG): container finished" podID="be00da51-8219-4cae-94b4-717851f75c79" containerID="fb4514fd05c488687648ce5efce753540d28890601dc6ac683914b3a479979df" exitCode=0
Sep 10 20:28:47.487707 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.487491    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"
Sep 10 20:28:47.487707 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.487555    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh" event={"ID":"be00da51-8219-4cae-94b4-717851f75c79","Type":"ContainerDied","Data":"fb4514fd05c488687648ce5efce753540d28890601dc6ac683914b3a479979df"}
Sep 10 20:28:47.487707 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.487599    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh" event={"ID":"be00da51-8219-4cae-94b4-717851f75c79","Type":"ContainerDied","Data":"ec1d8a507490dbea9e89f6b2b2daa0cd0071afd65f5e70886d5a3dbc2cdc4c25"}
Sep 10 20:28:47.487707 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.487631    2561 scope.go:117] "RemoveContainer" containerID="fb4514fd05c488687648ce5efce753540d28890601dc6ac683914b3a479979df"
Sep 10 20:28:47.497175 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.497157    2561 scope.go:117] "RemoveContainer" containerID="fb4514fd05c488687648ce5efce753540d28890601dc6ac683914b3a479979df"
Sep 10 20:28:47.497463 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:28:47.497445    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4514fd05c488687648ce5efce753540d28890601dc6ac683914b3a479979df\": container with ID starting with fb4514fd05c488687648ce5efce753540d28890601dc6ac683914b3a479979df not found: ID does not exist" containerID="fb4514fd05c488687648ce5efce753540d28890601dc6ac683914b3a479979df"
Sep 10 20:28:47.497556 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.497477    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4514fd05c488687648ce5efce753540d28890601dc6ac683914b3a479979df"} err="failed to get container status \"fb4514fd05c488687648ce5efce753540d28890601dc6ac683914b3a479979df\": rpc error: code = NotFound desc = could not find container \"fb4514fd05c488687648ce5efce753540d28890601dc6ac683914b3a479979df\": container with ID starting with fb4514fd05c488687648ce5efce753540d28890601dc6ac683914b3a479979df not found: ID does not exist"
Sep 10 20:28:47.504393 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.504367    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"]
Sep 10 20:28:47.507402 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:47.507381    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-15343da8-kserve-5d8f6bbddc-85qzh"]
Sep 10 20:28:49.403629 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:49.403580    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be00da51-8219-4cae-94b4-717851f75c79" path="/var/lib/kubelet/pods/be00da51-8219-4cae-94b4-717851f75c79/volumes"
Sep 10 20:28:53.013232 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.013198    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-559477f47-lgq7n"]
Sep 10 20:28:53.013724 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.013582    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd5df27-2616-4f3d-8acd-654c3b8c8f54" containerName="main"
Sep 10 20:28:53.013724 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.013596    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd5df27-2616-4f3d-8acd-654c3b8c8f54" containerName="main"
Sep 10 20:28:53.013724 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.013618    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be00da51-8219-4cae-94b4-717851f75c79" containerName="main"
Sep 10 20:28:53.013724 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.013624    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="be00da51-8219-4cae-94b4-717851f75c79" containerName="main"
Sep 10 20:28:53.013724 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.013677    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd5df27-2616-4f3d-8acd-654c3b8c8f54" containerName="main"
Sep 10 20:28:53.013724 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.013688    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="be00da51-8219-4cae-94b4-717851f75c79" containerName="main"
Sep 10 20:28:53.016521 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.016506    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:53.027005 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.026976    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-559477f47-lgq7n"]
Sep 10 20:28:53.089508 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.089468    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vljg\" (UniqueName: \"kubernetes.io/projected/3853cd29-da22-426b-9ffa-42549fb95738-kube-api-access-8vljg\") pod \"maas-api-559477f47-lgq7n\" (UID: \"3853cd29-da22-426b-9ffa-42549fb95738\") " pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:53.089701 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.089608    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3853cd29-da22-426b-9ffa-42549fb95738-metrics-certs\") pod \"maas-api-559477f47-lgq7n\" (UID: \"3853cd29-da22-426b-9ffa-42549fb95738\") " pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:53.089701 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.089639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3853cd29-da22-426b-9ffa-42549fb95738-maas-api-tls\") pod \"maas-api-559477f47-lgq7n\" (UID: \"3853cd29-da22-426b-9ffa-42549fb95738\") " pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:53.089701 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.089670    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3853cd29-da22-426b-9ffa-42549fb95738-openshift-service-ca\") pod \"maas-api-559477f47-lgq7n\" (UID: \"3853cd29-da22-426b-9ffa-42549fb95738\") " pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:53.190345 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.190310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3853cd29-da22-426b-9ffa-42549fb95738-metrics-certs\") pod \"maas-api-559477f47-lgq7n\" (UID: \"3853cd29-da22-426b-9ffa-42549fb95738\") " pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:53.190345 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.190348    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3853cd29-da22-426b-9ffa-42549fb95738-maas-api-tls\") pod \"maas-api-559477f47-lgq7n\" (UID: \"3853cd29-da22-426b-9ffa-42549fb95738\") " pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:53.190638 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.190372    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3853cd29-da22-426b-9ffa-42549fb95738-openshift-service-ca\") pod \"maas-api-559477f47-lgq7n\" (UID: \"3853cd29-da22-426b-9ffa-42549fb95738\") " pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:53.190638 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.190417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vljg\" (UniqueName: \"kubernetes.io/projected/3853cd29-da22-426b-9ffa-42549fb95738-kube-api-access-8vljg\") pod \"maas-api-559477f47-lgq7n\" (UID: \"3853cd29-da22-426b-9ffa-42549fb95738\") " pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:53.191086 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.191064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3853cd29-da22-426b-9ffa-42549fb95738-openshift-service-ca\") pod \"maas-api-559477f47-lgq7n\" (UID: \"3853cd29-da22-426b-9ffa-42549fb95738\") " pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:53.192914 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.192891    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3853cd29-da22-426b-9ffa-42549fb95738-metrics-certs\") pod \"maas-api-559477f47-lgq7n\" (UID: \"3853cd29-da22-426b-9ffa-42549fb95738\") " pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:53.193002 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.192940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3853cd29-da22-426b-9ffa-42549fb95738-maas-api-tls\") pod \"maas-api-559477f47-lgq7n\" (UID: \"3853cd29-da22-426b-9ffa-42549fb95738\") " pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:53.197985 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.197964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vljg\" (UniqueName: \"kubernetes.io/projected/3853cd29-da22-426b-9ffa-42549fb95738-kube-api-access-8vljg\") pod \"maas-api-559477f47-lgq7n\" (UID: \"3853cd29-da22-426b-9ffa-42549fb95738\") " pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:53.327311 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.327272    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:53.603484 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.603459    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-559477f47-lgq7n"]
Sep 10 20:28:53.605439 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:28:53.605416    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3853cd29_da22_426b_9ffa_42549fb95738.slice/crio-4668f9dc4373e3033a7a133fe84ae129731db2b189200cb613aba78e9f3f0c40 WatchSource:0}: Error finding container 4668f9dc4373e3033a7a133fe84ae129731db2b189200cb613aba78e9f3f0c40: Status 404 returned error can't find the container with id 4668f9dc4373e3033a7a133fe84ae129731db2b189200cb613aba78e9f3f0c40
Sep 10 20:28:53.606723 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:53.606705    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:28:54.534958 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:54.534926    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n" event={"ID":"3853cd29-da22-426b-9ffa-42549fb95738","Type":"ContainerStarted","Data":"f57b0e999412cd07708ff573622d13f5986a6e602aa1e1acfda658596645d4ef"}
Sep 10 20:28:54.535344 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:54.534977    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n" event={"ID":"3853cd29-da22-426b-9ffa-42549fb95738","Type":"ContainerStarted","Data":"4668f9dc4373e3033a7a133fe84ae129731db2b189200cb613aba78e9f3f0c40"}
Sep 10 20:28:54.535344 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:54.535055    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:28:54.552191 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:28:54.552146    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n" podStartSLOduration=2.115328621 podStartE2EDuration="2.55213225s" totalImagesPullingTime="436.803629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:28:54.551658965 +0000 UTC m=+2075.602186952" watchObservedRunningTime="2026-09-10 20:28:54.55213225 +0000 UTC m=+2075.602660238"
Sep 10 20:29:00.545919 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:00.545885    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-559477f47-lgq7n"
Sep 10 20:29:08.203427 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:08.203381    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"]
Sep 10 20:29:08.203946 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:08.203646    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc" podUID="f0cbd8c5-b68d-47dc-8339-d012dc5bfb65" containerName="maas-api" containerID="cri-o://0b8364da3843607b8c1408c1f544300d9908a9b8887edd18d8aeae7c83906710" gracePeriod=30
Sep 10 20:29:09.104732 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.104691    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc" podUID="f0cbd8c5-b68d-47dc-8339-d012dc5bfb65" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.38:8443/health\": dial tcp 10.134.0.38:8443: connect: connection refused"
Sep 10 20:29:09.442848 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.442818    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:29:09.533281 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.533242    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tnfl\" (UniqueName: \"kubernetes.io/projected/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-kube-api-access-4tnfl\") pod \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") "
Sep 10 20:29:09.533483 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.533305    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-metrics-certs\") pod \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") "
Sep 10 20:29:09.533483 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.533365    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-maas-api-tls\") pod \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") "
Sep 10 20:29:09.533483 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.533403    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-openshift-service-ca\") pod \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\" (UID: \"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65\") "
Sep 10 20:29:09.533832 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.533805    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f0cbd8c5-b68d-47dc-8339-d012dc5bfb65" (UID: "f0cbd8c5-b68d-47dc-8339-d012dc5bfb65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:29:09.535801 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.535763    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f0cbd8c5-b68d-47dc-8339-d012dc5bfb65" (UID: "f0cbd8c5-b68d-47dc-8339-d012dc5bfb65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:29:09.535801 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.535773    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-kube-api-access-4tnfl" (OuterVolumeSpecName: "kube-api-access-4tnfl") pod "f0cbd8c5-b68d-47dc-8339-d012dc5bfb65" (UID: "f0cbd8c5-b68d-47dc-8339-d012dc5bfb65"). InnerVolumeSpecName "kube-api-access-4tnfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:29:09.535940 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.535776    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f0cbd8c5-b68d-47dc-8339-d012dc5bfb65" (UID: "f0cbd8c5-b68d-47dc-8339-d012dc5bfb65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:29:09.617926 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.617836    2561 generic.go:356] "Generic (PLEG): container finished" podID="f0cbd8c5-b68d-47dc-8339-d012dc5bfb65" containerID="0b8364da3843607b8c1408c1f544300d9908a9b8887edd18d8aeae7c83906710" exitCode=0
Sep 10 20:29:09.617926 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.617879    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"
Sep 10 20:29:09.618116 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.617883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc" event={"ID":"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65","Type":"ContainerDied","Data":"0b8364da3843607b8c1408c1f544300d9908a9b8887edd18d8aeae7c83906710"}
Sep 10 20:29:09.618116 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.618014    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc" event={"ID":"f0cbd8c5-b68d-47dc-8339-d012dc5bfb65","Type":"ContainerDied","Data":"424e9bb9aac05e2b92ca7ebdb8cf10af537bf7c690d5b111587352b2b22f402c"}
Sep 10 20:29:09.618116 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.618047    2561 scope.go:117] "RemoveContainer" containerID="0b8364da3843607b8c1408c1f544300d9908a9b8887edd18d8aeae7c83906710"
Sep 10 20:29:09.627361 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.627342    2561 scope.go:117] "RemoveContainer" containerID="0b8364da3843607b8c1408c1f544300d9908a9b8887edd18d8aeae7c83906710"
Sep 10 20:29:09.627706 ip-10-0-140-111 kubenswrapper[2561]: E0910 20:29:09.627684    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8364da3843607b8c1408c1f544300d9908a9b8887edd18d8aeae7c83906710\": container with ID starting with 0b8364da3843607b8c1408c1f544300d9908a9b8887edd18d8aeae7c83906710 not found: ID does not exist" containerID="0b8364da3843607b8c1408c1f544300d9908a9b8887edd18d8aeae7c83906710"
Sep 10 20:29:09.627769 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.627717    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8364da3843607b8c1408c1f544300d9908a9b8887edd18d8aeae7c83906710"} err="failed to get container status \"0b8364da3843607b8c1408c1f544300d9908a9b8887edd18d8aeae7c83906710\": rpc error: code = NotFound desc = could not find container \"0b8364da3843607b8c1408c1f544300d9908a9b8887edd18d8aeae7c83906710\": container with ID starting with 0b8364da3843607b8c1408c1f544300d9908a9b8887edd18d8aeae7c83906710 not found: ID does not exist"
Sep 10 20:29:09.634315 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.634289    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:29:09.634315 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.634315    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tnfl\" (UniqueName: \"kubernetes.io/projected/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-kube-api-access-4tnfl\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:29:09.634443 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.634324    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-metrics-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:29:09.634443 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.634333    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\""
Sep 10 20:29:09.641468 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.641430    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"]
Sep 10 20:29:09.643473 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:09.643452    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7f8945f9-qt2rc"]
Sep 10 20:29:11.404314 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:11.404276    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0cbd8c5-b68d-47dc-8339-d012dc5bfb65" path="/var/lib/kubelet/pods/f0cbd8c5-b68d-47dc-8339-d012dc5bfb65/volumes"
Sep 10 20:29:20.954829 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:20.954716    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 20:29:20.978666 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:20.969267    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 20:29:28.824181 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:28.824148    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-57hqs_810e302d-18ea-4e8d-b0c3-68c974aa7cc0/server/0.log"
Sep 10 20:29:28.955346 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:28.955310    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-f8hnl_84d2936e-2f1f-4334-a600-4dc6e33172d5/manager/0.log"
Sep 10 20:29:29.082012 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:29.081929    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5678c4c4c5-hvgs9_b9d14710-c876-4054-9773-93ef56c10bf4/copy-manifests/0.log"
Sep 10 20:29:29.089749 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:29.089704    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5678c4c4c5-hvgs9_b9d14710-c876-4054-9773-93ef56c10bf4/manager/0.log"
Sep 10 20:29:29.822667 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:29.822638    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-559477f47-lgq7n_3853cd29-da22-426b-9ffa-42549fb95738/maas-api/0.log"
Sep 10 20:29:32.175950 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:32.175916    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576c9858ff-hknzf_7bb2c398-ca3f-4378-9eb9-93ce9bac8db5/kube-auth-proxy/0.log"
Sep 10 20:29:33.626725 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:33.626689    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft52f6_3a6f9114-2eff-4720-83a6-f98263aaabd4/main/0.log"
Sep 10 20:29:33.861975 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:33.861943    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-g58br_08530dcd-3b4c-4b11-b6af-32898f41fc84/main/0.log"
Sep 10 20:29:39.394492 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:39.394445    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:29:39.881002 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:39.880966    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g28c4/must-gather-7jqrp"]
Sep 10 20:29:39.881312 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:39.881300    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0cbd8c5-b68d-47dc-8339-d012dc5bfb65" containerName="maas-api"
Sep 10 20:29:39.881358 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:39.881314    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0cbd8c5-b68d-47dc-8339-d012dc5bfb65" containerName="maas-api"
Sep 10 20:29:39.881392 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:39.881382    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0cbd8c5-b68d-47dc-8339-d012dc5bfb65" containerName="maas-api"
Sep 10 20:29:39.884462 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:39.884441    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g28c4/must-gather-7jqrp"
Sep 10 20:29:39.887808 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:39.887775    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g28c4\"/\"kube-root-ca.crt\""
Sep 10 20:29:39.888053 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:39.888037    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g28c4\"/\"openshift-service-ca.crt\""
Sep 10 20:29:39.888151 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:39.888063    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g28c4\"/\"default-dockercfg-qnx6f\""
Sep 10 20:29:39.903643 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:39.903601    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g28c4/must-gather-7jqrp"]
Sep 10 20:29:39.910291 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:39.910263    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:29:39.998148 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:39.998116    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxhwd\" (UniqueName: \"kubernetes.io/projected/cb30efd6-762a-4e92-9816-11558580adbf-kube-api-access-lxhwd\") pod \"must-gather-7jqrp\" (UID: \"cb30efd6-762a-4e92-9816-11558580adbf\") " pod="openshift-must-gather-g28c4/must-gather-7jqrp"
Sep 10 20:29:39.998349 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:39.998178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb30efd6-762a-4e92-9816-11558580adbf-must-gather-output\") pod \"must-gather-7jqrp\" (UID: \"cb30efd6-762a-4e92-9816-11558580adbf\") " pod="openshift-must-gather-g28c4/must-gather-7jqrp"
Sep 10 20:29:39.999033 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:39.999006    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log" limit=1024
Sep 10 20:29:40.099241 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:40.099202    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb30efd6-762a-4e92-9816-11558580adbf-must-gather-output\") pod \"must-gather-7jqrp\" (UID: \"cb30efd6-762a-4e92-9816-11558580adbf\") " pod="openshift-must-gather-g28c4/must-gather-7jqrp"
Sep 10 20:29:40.099442 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:40.099295    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhwd\" (UniqueName: \"kubernetes.io/projected/cb30efd6-762a-4e92-9816-11558580adbf-kube-api-access-lxhwd\") pod \"must-gather-7jqrp\" (UID: \"cb30efd6-762a-4e92-9816-11558580adbf\") " pod="openshift-must-gather-g28c4/must-gather-7jqrp"
Sep 10 20:29:40.099638 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:40.099609    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb30efd6-762a-4e92-9816-11558580adbf-must-gather-output\") pod \"must-gather-7jqrp\" (UID: \"cb30efd6-762a-4e92-9816-11558580adbf\") " pod="openshift-must-gather-g28c4/must-gather-7jqrp"
Sep 10 20:29:40.107823 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:40.107797    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxhwd\" (UniqueName: \"kubernetes.io/projected/cb30efd6-762a-4e92-9816-11558580adbf-kube-api-access-lxhwd\") pod \"must-gather-7jqrp\" (UID: \"cb30efd6-762a-4e92-9816-11558580adbf\") " pod="openshift-must-gather-g28c4/must-gather-7jqrp"
Sep 10 20:29:40.193923 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:40.193836    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g28c4/must-gather-7jqrp"
Sep 10 20:29:40.473846 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:40.473820    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g28c4/must-gather-7jqrp"]
Sep 10 20:29:40.475557 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:29:40.475499    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb30efd6_762a_4e92_9816_11558580adbf.slice/crio-7877f9ec0747148b7742cd46fc49c06c02f6330103aca3fe8bcce1650cebe852 WatchSource:0}: Error finding container 7877f9ec0747148b7742cd46fc49c06c02f6330103aca3fe8bcce1650cebe852: Status 404 returned error can't find the container with id 7877f9ec0747148b7742cd46fc49c06c02f6330103aca3fe8bcce1650cebe852
Sep 10 20:29:40.783290 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:40.783257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g28c4/must-gather-7jqrp" event={"ID":"cb30efd6-762a-4e92-9816-11558580adbf","Type":"ContainerStarted","Data":"7877f9ec0747148b7742cd46fc49c06c02f6330103aca3fe8bcce1650cebe852"}
Sep 10 20:29:41.795401 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:41.794294    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g28c4/must-gather-7jqrp" event={"ID":"cb30efd6-762a-4e92-9816-11558580adbf","Type":"ContainerStarted","Data":"1c0c91ca77fd540f8d94d8ca6e410c8ec9b42dde5972ed8b5febae98f9c2d36b"}
Sep 10 20:29:41.795401 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:41.794349    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g28c4/must-gather-7jqrp" event={"ID":"cb30efd6-762a-4e92-9816-11558580adbf","Type":"ContainerStarted","Data":"61d50222a62d489859e3b98680679578b6c10fd0c1ddd9c6c8ef69afba1007ea"}
Sep 10 20:29:41.820502 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:41.820432    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g28c4/must-gather-7jqrp" podStartSLOduration=1.943800343 podStartE2EDuration="2.820412257s" totalImagesPullingTime="876.611914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:29:41.816220606 +0000 UTC m=+2122.866748599" watchObservedRunningTime="2026-09-10 20:29:41.820412257 +0000 UTC m=+2122.870940246"
Sep 10 20:29:43.144545 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:43.144501    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tvxpw_c51fb2f9-4a5c-4259-8087-67560d10732b/global-pull-secret-syncer/0.log"
Sep 10 20:29:43.209592 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:43.209557    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gdv2q_85d11857-2f7e-4bae-a7eb-dc6fe5a0b527/konnectivity-agent/0.log"
Sep 10 20:29:43.404430 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:43.404336    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-111.ec2.internal_10643e86bc3da5fa40e6977a1909b44c/haproxy/0.log"
Sep 10 20:29:50.546773 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:50.546740    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-lpbmp_5f652bf6-f124-4f0f-a594-773e4bfed694/cluster-monitoring-operator/0.log"
Sep 10 20:29:50.863128 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:50.863009    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kg9km_f2549669-329a-42c7-bcd9-72f7430bb367/node-exporter/0.log"
Sep 10 20:29:50.891057 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:50.891026    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kg9km_f2549669-329a-42c7-bcd9-72f7430bb367/kube-rbac-proxy/0.log"
Sep 10 20:29:50.917853 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:50.917819    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kg9km_f2549669-329a-42c7-bcd9-72f7430bb367/init-textfile/0.log"
Sep 10 20:29:51.303515 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:51.303478    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-dvhwx_e777acf3-d8a1-42ca-88ef-5c1cda860bdd/prometheus-operator/0.log"
Sep 10 20:29:51.324273 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:51.324230    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-dvhwx_e777acf3-d8a1-42ca-88ef-5c1cda860bdd/kube-rbac-proxy/0.log"
Sep 10 20:29:51.474785 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:51.474751    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-864dd47945-42tdf_78da9b4e-ffd2-43ad-b8d7-95f3421aa71a/thanos-query/0.log"
Sep 10 20:29:51.499473 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:51.499439    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-864dd47945-42tdf_78da9b4e-ffd2-43ad-b8d7-95f3421aa71a/kube-rbac-proxy-web/0.log"
Sep 10 20:29:51.527223 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:51.527196    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-864dd47945-42tdf_78da9b4e-ffd2-43ad-b8d7-95f3421aa71a/kube-rbac-proxy/0.log"
Sep 10 20:29:51.549974 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:51.549947    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-864dd47945-42tdf_78da9b4e-ffd2-43ad-b8d7-95f3421aa71a/prom-label-proxy/0.log"
Sep 10 20:29:51.572498 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:51.572413    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-864dd47945-42tdf_78da9b4e-ffd2-43ad-b8d7-95f3421aa71a/kube-rbac-proxy-rules/0.log"
Sep 10 20:29:51.598586 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:51.598551    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-864dd47945-42tdf_78da9b4e-ffd2-43ad-b8d7-95f3421aa71a/kube-rbac-proxy-metrics/0.log"
Sep 10 20:29:53.106025 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.105981    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"]
Sep 10 20:29:53.109658 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.109622    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.125757 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.125718    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"]
Sep 10 20:29:53.236207 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.236173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6aa7915-4e6c-4151-93f2-3a759dd59c96-sys\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.236207 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.236212    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6aa7915-4e6c-4151-93f2-3a759dd59c96-lib-modules\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.236478 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.236272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6aa7915-4e6c-4151-93f2-3a759dd59c96-podres\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.236478 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.236319    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgptm\" (UniqueName: \"kubernetes.io/projected/b6aa7915-4e6c-4151-93f2-3a759dd59c96-kube-api-access-vgptm\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.236478 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.236343    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6aa7915-4e6c-4151-93f2-3a759dd59c96-proc\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.337481 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.337438    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6aa7915-4e6c-4151-93f2-3a759dd59c96-sys\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.337696 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.337505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6aa7915-4e6c-4151-93f2-3a759dd59c96-lib-modules\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.337696 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.337610    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6aa7915-4e6c-4151-93f2-3a759dd59c96-podres\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.337696 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.337671    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgptm\" (UniqueName: \"kubernetes.io/projected/b6aa7915-4e6c-4151-93f2-3a759dd59c96-kube-api-access-vgptm\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.337878 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.337715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6aa7915-4e6c-4151-93f2-3a759dd59c96-proc\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.337878 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.337840    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6aa7915-4e6c-4151-93f2-3a759dd59c96-proc\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.337991 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.337902    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6aa7915-4e6c-4151-93f2-3a759dd59c96-sys\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.338049 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.338024    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6aa7915-4e6c-4151-93f2-3a759dd59c96-lib-modules\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.338128 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.338104    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6aa7915-4e6c-4151-93f2-3a759dd59c96-podres\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.348563 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.347994    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgptm\" (UniqueName: \"kubernetes.io/projected/b6aa7915-4e6c-4151-93f2-3a759dd59c96-kube-api-access-vgptm\") pod \"perf-node-gather-daemonset-h8cbs\" (UID: \"b6aa7915-4e6c-4151-93f2-3a759dd59c96\") " pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.426873 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.426782    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.574993 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.574957    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/1.log"
Sep 10 20:29:53.585949 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.585913    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-f7dn5_d7ac6a59-db3b-40a2-95f0-42f55a9156d1/console-operator/2.log"
Sep 10 20:29:53.731677 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.731605    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"]
Sep 10 20:29:53.733521 ip-10-0-140-111 kubenswrapper[2561]: W0910 20:29:53.733477    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb6aa7915_4e6c_4151_93f2_3a759dd59c96.slice/crio-0f3c43908ccfb69c27f72e6559cb137ba1fb64c9f065ae5c70c51e80dac44705 WatchSource:0}: Error finding container 0f3c43908ccfb69c27f72e6559cb137ba1fb64c9f065ae5c70c51e80dac44705: Status 404 returned error can't find the container with id 0f3c43908ccfb69c27f72e6559cb137ba1fb64c9f065ae5c70c51e80dac44705
Sep 10 20:29:53.912201 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.912166    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs" event={"ID":"b6aa7915-4e6c-4151-93f2-3a759dd59c96","Type":"ContainerStarted","Data":"d6f5d65e2949d79696c230fe14de72c4223dfe8c0497d1df9fdfaa156aa639db"}
Sep 10 20:29:53.912385 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.912228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs" event={"ID":"b6aa7915-4e6c-4151-93f2-3a759dd59c96","Type":"ContainerStarted","Data":"0f3c43908ccfb69c27f72e6559cb137ba1fb64c9f065ae5c70c51e80dac44705"}
Sep 10 20:29:53.912385 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.912256    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:29:53.937142 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:53.937076    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs" podStartSLOduration=0.937051108 podStartE2EDuration="937.051108ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:29:53.933986859 +0000 UTC m=+2134.984514848" watchObservedRunningTime="2026-09-10 20:29:53.937051108 +0000 UTC m=+2134.987579098"
Sep 10 20:29:56.111739 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:56.111695    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6lsgj_34f0063f-49f4-483d-9d24-acd1be984aee/dns/0.log"
Sep 10 20:29:56.133754 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:56.133710    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6lsgj_34f0063f-49f4-483d-9d24-acd1be984aee/kube-rbac-proxy/0.log"
Sep 10 20:29:56.274800 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:56.274766    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tb2sv_0e80f59a-083f-401d-92ba-ae440fb4840e/dns-node-resolver/0.log"
Sep 10 20:29:56.872721 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:56.872682    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-577c58467f-zwvr7_cadeb1ce-a05e-49ad-870b-9b4afb130bfe/registry/0.log"
Sep 10 20:29:56.923685 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:56.923655    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tmzvv_2d6179f7-5886-457f-9b86-a9849a6f3eaa/node-ca/0.log"
Sep 10 20:29:58.354003 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:58.353974    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576c9858ff-hknzf_7bb2c398-ca3f-4378-9eb9-93ce9bac8db5/kube-auth-proxy/0.log"
Sep 10 20:29:59.501272 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:59.501242    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t4hc6_4ae9e838-42d8-479a-a2c3-fd07710f2a34/serve-healthcheck-canary/0.log"
Sep 10 20:29:59.932508 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:29:59.932469    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g28c4/perf-node-gather-daemonset-h8cbs"
Sep 10 20:30:00.005998 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:00.005966    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-vnjtm_7c8de1db-4a70-469e-8f56-fecd54dde30a/insights-operator/0.log"
Sep 10 20:30:00.006822 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:00.006795    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-vnjtm_7c8de1db-4a70-469e-8f56-fecd54dde30a/insights-operator/1.log"
Sep 10 20:30:00.205064 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:00.204976    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z7mxd_4433866a-197a-4fdf-abc5-6c575d5eea05/kube-rbac-proxy/0.log"
Sep 10 20:30:00.231775 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:00.231742    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z7mxd_4433866a-197a-4fdf-abc5-6c575d5eea05/exporter/0.log"
Sep 10 20:30:00.258894 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:00.258862    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z7mxd_4433866a-197a-4fdf-abc5-6c575d5eea05/extractor/0.log"
Sep 10 20:30:03.175690 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:03.175658    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-57hqs_810e302d-18ea-4e8d-b0c3-68c974aa7cc0/server/0.log"
Sep 10 20:30:03.293856 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:03.293802    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-f8hnl_84d2936e-2f1f-4334-a600-4dc6e33172d5/manager/0.log"
Sep 10 20:30:03.356309 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:03.356218    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5678c4c4c5-hvgs9_b9d14710-c876-4054-9773-93ef56c10bf4/manager/0.log"
Sep 10 20:30:03.438123 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:03.438091    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5678c4c4c5-hvgs9_b9d14710-c876-4054-9773-93ef56c10bf4/copy-manifests/0.log"
Sep 10 20:30:04.695170 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:04.695128    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576c9858ff-hknzf_7bb2c398-ca3f-4378-9eb9-93ce9bac8db5/kube-auth-proxy/0.log"
Sep 10 20:30:05.098201 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:05.098168    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6fd976b7bd-qkcsb_738906e4-a1a1-4db4-8de1-2ce79e3a548f/manager/0.log"
Sep 10 20:30:05.134401 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:05.134366    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-8jsw2_2b27fc61-c745-4948-b94e-2a5c1c393a9a/openshift-lws-operator/0.log"
Sep 10 20:30:06.838926 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:06.838887    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576c9858ff-hknzf_7bb2c398-ca3f-4378-9eb9-93ce9bac8db5/kube-auth-proxy/0.log"
Sep 10 20:30:10.494980 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:10.494898    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-72k6n_bd7ca140-2403-42d3-92b8-df81800237d3/migrator/0.log"
Sep 10 20:30:10.522868 ip-10-0-140-111 kubenswrapper[2561]: I0910 20:30:10.522843    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-72k6n_bd7ca140-2403-42d3-92b8-df81800237d3/graceful-termination/0.log"