Sep 12 00:41:23.381221 ip-10-0-139-121 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 12 00:41:23.381232 ip-10-0-139-121 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 12 00:41:23.381242 ip-10-0-139-121 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 12 00:41:23.381555 ip-10-0-139-121 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 12 00:41:33.452453 ip-10-0-139-121 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 12 00:41:33.452470 ip-10-0-139-121 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5dfeeb2244eb43c3aa53a8af6f9aaaae --
Sep 12 00:43:15.331236 ip-10-0-139-121 systemd[1]: Starting Kubernetes Kubelet...
Sep 12 00:43:15.720867 ip-10-0-139-121 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 00:43:15.720867 ip-10-0-139-121 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 12 00:43:15.720867 ip-10-0-139-121 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 00:43:15.720867 ip-10-0-139-121 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 12 00:43:15.720867 ip-10-0-139-121 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 00:43:15.723918 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.723826    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 12 00:43:15.727358 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727334    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:43:15.727358 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727353    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:43:15.727358 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727357    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:43:15.727358 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727361    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727368    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727372    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727376    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727380    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727384    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727388    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727392    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727396    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727400    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727404    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727407    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727411    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727414    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727419    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727423    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727428    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727432    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727436    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727440    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:43:15.727584 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727444    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727448    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727452    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727455    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727460    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727464    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727468    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727471    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727476    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727480    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727484    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727489    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727493    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727497    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727502    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727506    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727510    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727514    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727518    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727523    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:43:15.728357 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727527    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727531    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727536    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727540    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727545    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727549    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727553    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727616    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727669    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727674    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727678    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727684    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727688    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727696    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727702    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727707    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727711    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727716    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727720    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:43:15.728986 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727725    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727731    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727839    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727863    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727870    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727875    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727880    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727889    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727898    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727904    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727909    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727914    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727919    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727923    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727935    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727939    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727966    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727973    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:43:15.729441 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727978    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727982    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727987    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727992    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.727996    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.728001    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.728006    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.728016    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.728021    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.728025    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.728029    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.728034    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.728038    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.728042    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.728046    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.728050    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.728055    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.728062    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.728067    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:43:15.729883 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729403    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729415    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729418    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729422    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729424    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729428    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729431    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729434    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729436    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729440    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729443    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729446    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729448    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729451    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729454    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729457    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729459    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729462    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729465    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729468    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:43:15.730361 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729471    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729473    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729476    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729478    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729481    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729486    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729490    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729493    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729495    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729498    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729500    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729503    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729505    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729509    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729512    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729514    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729517    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729519    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729522    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:43:15.730850 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729524    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729527    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729529    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729532    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729534    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729537    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729539    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729542    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729544    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729547    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729549    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729552    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729555    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729557    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729559    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729562    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729564    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729566    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729569    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:43:15.731330 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729571    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729574    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729576    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729578    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729581    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729583    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729585    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729589    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729593    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729597    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729600    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729603    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729607    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729609    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729612    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729614    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729617    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729621    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729624    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:43:15.731798 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729627    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729630    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729632    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729635    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729637    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729639    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729642    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729644    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729647    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729649    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729652    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729654    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729657    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729659    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729661    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729664    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729666    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729669    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729671    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729673    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:43:15.732308 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729677    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.729679    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730090    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730096    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730098    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730102    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730104    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730107    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730110    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730112    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730115    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730118    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730120    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730123    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730125    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730128    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730130    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730133    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730135    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730138    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:43:15.732810 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730141    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730143    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730146    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730148    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730151    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730153    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730155    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730158    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730160    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730163    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730165    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730168    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730171    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730174    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730176    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730179    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730181    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730185    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730189    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730193    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:43:15.733310 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730195    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730198    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730201    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730204    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730207    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730209    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730212    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730215    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730218    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730220    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730223    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730225    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730227    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730230    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730232    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730235    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730237    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730240    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730242    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730245    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:43:15.733786 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730247    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730250    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730252    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730255    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730257    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730260    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730262    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730265    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730270    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730274    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730277    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730280    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730284    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730287    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730290    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730293    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730295    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730298    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:43:15.734313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730301    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730303    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730306    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730308    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730310    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730313    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730315    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730317    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730320    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730322    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730325    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730328    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730330    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730333    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730335    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730338    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730340    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730343    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730345    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730347    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:43:15.734750 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730350    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730352    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.730354    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730425    2567 flags.go:64] FLAG: --address="0.0.0.0"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730461    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730467    2567 flags.go:64] FLAG: --anonymous-auth="true"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730471    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730475    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730478    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730482    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730488    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730491    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730494    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730497    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730500    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730503    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730506    2567 flags.go:64] FLAG: --cgroup-root=""
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730509    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730511    2567 flags.go:64] FLAG: --client-ca-file=""
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730514    2567 flags.go:64] FLAG: --cloud-provider="external"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730517    2567 flags.go:64] FLAG: --cluster-dns="[]"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730521    2567 flags.go:64] FLAG: --cluster-domain=""
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730526    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730529    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 12 00:43:15.735235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730532    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730536    2567 flags.go:64] FLAG: --container-log-max-files="5"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730540    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730543    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730546    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730549    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730552    2567 flags.go:64] FLAG: --contention-profiling="false"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730555    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730558    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730561    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730564    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730567    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730571    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730573    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730576    2567 flags.go:64] FLAG: --enable-load-reader="false"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730579    2567 flags.go:64] FLAG: --enable-server="true"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730582    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730585    2567 flags.go:64] FLAG: --event-burst="100"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730588    2567 flags.go:64] FLAG: --event-qps="50"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730591    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730594    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730597    2567 flags.go:64] FLAG: --eviction-hard=""
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730600    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730603    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730606    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 12 00:43:15.735840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730609    2567 flags.go:64] FLAG: --eviction-soft=""
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730611    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730614    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730617    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730620    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730624    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730627    2567 flags.go:64] FLAG: --fail-swap-on="true"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730630    2567 flags.go:64] FLAG: --feature-gates=""
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730633    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730636    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730639    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730642    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730645    2567 flags.go:64] FLAG: --healthz-port="10248"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730648    2567 flags.go:64] FLAG: --help="false"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730651    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-139-121.ec2.internal"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730654    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730657    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730660    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730663    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730666    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730669    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730673    2567 flags.go:64] FLAG: --image-service-endpoint=""
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730675    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 12 00:43:15.736507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730678    2567 flags.go:64] FLAG: --kube-api-burst="100"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730681    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730684    2567 flags.go:64] FLAG: --kube-api-qps="50"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730687    2567 flags.go:64] FLAG: --kube-reserved=""
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730690    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730692    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730695    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730698    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730701    2567 flags.go:64] FLAG: --lock-file=""
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730703    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730706    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730709    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730716    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730718    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730723    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730726    2567 flags.go:64] FLAG: --logging-format="text"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730728    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730732    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730734    2567 flags.go:64] FLAG: --manifest-url=""
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730737    2567 flags.go:64] FLAG: --manifest-url-header=""
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730742    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730744    2567 flags.go:64] FLAG: --max-open-files="1000000"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730748    2567 flags.go:64] FLAG: --max-pods="110"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730751    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730754    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 12 00:43:15.737110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730757    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730759    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730762    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730765    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730768    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730776    2567 flags.go:64] FLAG: --node-status-max-images="50"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730779    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730781    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730784    2567 flags.go:64] FLAG: --pod-cidr=""
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730787    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730791    2567 flags.go:64] FLAG: --pod-manifest-path=""
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730795    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730797    2567 flags.go:64] FLAG: --pods-per-core="0"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730800    2567 flags.go:64] FLAG: --port="10250"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730803    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730806    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0882972df4313377b"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730809    2567 flags.go:64] FLAG: --qos-reserved=""
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730812    2567 flags.go:64] FLAG: --read-only-port="10255"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730814    2567 flags.go:64] FLAG: --register-node="true"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730819    2567 flags.go:64] FLAG: --register-with-taints=""
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730822    2567 flags.go:64] FLAG: --registry-burst="10"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730825    2567 flags.go:64] FLAG: --registry-qps="5"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730829    2567 flags.go:64] FLAG: --reserved-cpus=""
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730832    2567 flags.go:64] FLAG: --reserved-memory=""
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730835    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 12 00:43:15.737718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730838    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730841    2567 flags.go:64] FLAG: --rotate-certificates="false"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730844    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730847    2567 flags.go:64] FLAG: --runonce="false"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730849    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730852    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730856    2567 flags.go:64] FLAG: --seccomp-default="false"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730858    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730861    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730865    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730868    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730870    2567 flags.go:64] FLAG: --storage-driver-password="root"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730873    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730877    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730880    2567 flags.go:64] FLAG: --storage-driver-user="root"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730882    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730885    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730888    2567 flags.go:64] FLAG: --system-cgroups=""
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730891    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730895    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730898    2567 flags.go:64] FLAG: --tls-cert-file=""
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730901    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730908    2567 flags.go:64] FLAG: --tls-min-version=""
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730911    2567 flags.go:64] FLAG: --tls-private-key-file=""
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730914    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 12 00:43:15.738351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730916    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730920    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730924    2567 flags.go:64] FLAG: --v="2"
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730927    2567 flags.go:64] FLAG: --version="false"
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730933    2567 flags.go:64] FLAG: --vmodule=""
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730936    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.730939    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731069    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731074    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731077    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731080    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731082    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731085    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731088    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731091    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731093    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731096    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731098    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731101    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731103    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731105    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:43:15.738982 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731114    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731116    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731119    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731121    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731124    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731126    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731129    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731132    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731134    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731137    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731139    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731142    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731145    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731148    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731151    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731156    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731158    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731161    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731163    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731166    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:43:15.739535 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731168    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731171    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731173    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731176    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731178    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731181    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731184    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731186    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731189    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731192    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731194    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731196    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731199    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731208    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731211    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731213    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731216    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731218    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731221    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731223    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:43:15.740028 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731226    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731228    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731231    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731233    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731237    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731240    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731243    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731248    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731252    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731255    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731258    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731260    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731263    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731266    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731268    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731271    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731273    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731276    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731278    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:43:15.740514 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731282    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731286    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731290    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731293    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731296    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731299    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731301    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731307    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731309    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731312    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731314    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731317    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731319    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731322    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731324    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731326    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731329    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731332    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731335    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:43:15.740994 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731338    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731342    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731344    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731347    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731349    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731352    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.731354    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.732228    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.740175    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.740190    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.741071    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.741079    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741144    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741149    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741152    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:43:15.741448 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741155    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741158    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741161    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741164    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741166    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741169    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741171    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741174    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741177    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741180    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741182    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741185    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741188    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741190    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741193    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741196    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741198    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741201    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741203    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741206    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:43:15.741825 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741208    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741212    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741216    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741219    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741221    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741224    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741229    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741231    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741234    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741236    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741239    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741241    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741244    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741246    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741248    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741251    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741253    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741256    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741258    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741261    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:43:15.742313 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741263    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741265    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741268    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741270    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741273    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741275    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741278    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741280    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741283    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741285    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741288    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741290    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741292    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741295    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741297    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741299    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741302    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741304    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741308    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:43:15.742934 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741311    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741313    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741316    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741318    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741321    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741323    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741326    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741328    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741330    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741333    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741335    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741337    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741340    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741342    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741345    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741347    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741349    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741352    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741354    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741357    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:43:15.743414 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741360    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741362    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741366    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741369    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741372    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741375    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741377    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741379    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741382    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741384    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741387    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741390    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741392    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741395    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741398    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741400    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:43:15.743890 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741404    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.741409    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741523    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741528    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741531    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741534    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741537    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741540    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741543    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741546    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741549    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741551    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741554    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741557    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741559    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:43:15.744306 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741562    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741564    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741568    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741572    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741575    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741578    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741581    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741583    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741586    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741589    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741592    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741595    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741597    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741600    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741603    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741606    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741609    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741611    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741614    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:43:15.744675 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741617    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741619    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741621    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741624    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741626    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741629    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741631    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741634    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741636    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741639    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741641    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741643    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741646    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741648    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741651    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741654    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741657    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741660    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741663    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741665    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:43:15.745152 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741668    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741670    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741673    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741675    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741677    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741680    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741683    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741685    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741688    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741691    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741693    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741696    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741698    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741701    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741703    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741705    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741708    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741710    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741712    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:43:15.745625 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741716    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741720    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741722    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741726    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741728    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741731    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741734    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741737    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741739    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741742    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741744    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741747    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741749    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741752    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741754    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741756    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741759    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741762    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741764    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741767    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:43:15.746101 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741770    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:43:15.746572 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741772    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:43:15.746572 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741774    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:43:15.746572 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741777    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:43:15.746572 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741780    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:43:15.746572 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741782    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:43:15.746572 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741785    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:43:15.746572 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:15.741787    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:43:15.746572 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.741792    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 00:43:15.746572 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.741905    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 12 00:43:15.746572 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.745363    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 12 00:43:15.746572 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.746417    2567 server.go:1019] "Starting client certificate rotation"
Sep 12 00:43:15.746572 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.746521    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 12 00:43:15.746859 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.746578    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 12 00:43:15.749299 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.749285    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 12 00:43:15.755671 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.755652    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 12 00:43:15.769047 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.769029    2567 log.go:25] "Validated CRI v1 runtime API"
Sep 12 00:43:15.772341 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.772322    2567 log.go:25] "Validated CRI v1 image API"
Sep 12 00:43:15.773197 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.773181    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 12 00:43:15.779434 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.779411    2567 fs.go:135] Filesystem UUIDs: map[069b6c5c-6ade-4c20-8dac-a0677f7e7d3f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 dce57123-48d5-4c31-8e5d-90b3d4df7161:/dev/nvme0n1p4]
Sep 12 00:43:15.779495 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.779434    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 12 00:43:15.780029 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.780009    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 12 00:43:15.785527 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.785412    2567 manager.go:216] Machine: {Timestamp:2026-09-12 00:43:15.784143124 +0000 UTC m=+0.344594795 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092463 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c240dbfd22b4fc383e2cae21cfe47 SystemUUID:ec2c240d-bfd2-2b4f-c383-e2cae21cfe47 BootID:5dfeeb22-44eb-43c3-aa53-a8af6f9aaaae Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:95:d3:06:7b:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:95:d3:06:7b:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:9f:40:c5:2e:af Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 12 00:43:15.785527 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.785526    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 12 00:43:15.785637 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.785612    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 12 00:43:15.786840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.786815    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 12 00:43:15.787001 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.786843    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-121.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 12 00:43:15.788383 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.788372    2567 topology_manager.go:138] "Creating topology manager with none policy"
Sep 12 00:43:15.788424 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.788385    2567 container_manager_linux.go:309] "Creating device plugin manager"
Sep 12 00:43:15.788424 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.788394    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 12 00:43:15.789233 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.789223    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 12 00:43:15.790437 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.790427    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 12 00:43:15.795794 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.795782    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 12 00:43:15.795905 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.795896    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 12 00:43:15.795938 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.795926    2567 kubelet.go:486] "Attempting to sync node with API server"
Sep 12 00:43:15.795938 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.795936    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 12 00:43:15.796006 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.795964    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 12 00:43:15.796006 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.795976    2567 kubelet.go:398] "Adding apiserver pod source"
Sep 12 00:43:15.796006 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.795984    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 12 00:43:15.796318 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.796306    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 12 00:43:15.796370 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.796331    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 12 00:43:15.798590 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.798574    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 12 00:43:15.801640 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.801624    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 12 00:43:15.801719 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.801662    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 12 00:43:15.803622 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.803608    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 12 00:43:15.803699 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.803627    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 12 00:43:15.803699 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.803636    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 12 00:43:15.803699 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.803644    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 12 00:43:15.803699 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.803655    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 12 00:43:15.803699 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.803663    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 12 00:43:15.803699 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.803683    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 12 00:43:15.803699 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.803691    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 12 00:43:15.803933 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.803717    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 12 00:43:15.803933 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.803727    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 12 00:43:15.803933 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.803751    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 12 00:43:15.803933 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.803759    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 12 00:43:15.804650 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.804638    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 12 00:43:15.804702 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.804654    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 12 00:43:15.808124 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.808106    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-121.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 12 00:43:15.808187 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:15.808158    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 12 00:43:15.808361 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:15.808342    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 12 00:43:15.808661 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.808647    2567 server.go:1268] "Started kubelet"
Sep 12 00:43:15.808830 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.808753    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 12 00:43:15.808922 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.808850    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 12 00:43:15.816974 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.809085    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 12 00:43:15.817297 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.817276    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 12 00:43:15.817410 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.817389    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x2hz8"
Sep 12 00:43:15.818008 ip-10-0-139-121 systemd[1]: Started Kubernetes Kubelet.
Sep 12 00:43:15.820209 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.820189    2567 server.go:310] "Adding debug handlers to kubelet server"
Sep 12 00:43:15.821448 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.821427    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x2hz8"
Sep 12 00:43:15.821910 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:15.821897    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 12 00:43:15.825716 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:15.824637    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-121.ec2.internal.18d46c167c9c2248  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-121.ec2.internal,UID:ip-10-0-139-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-121.ec2.internal,},FirstTimestamp:2026-09-12 00:43:15.808617032 +0000 UTC m=+0.369068691,LastTimestamp:2026-09-12 00:43:15.808617032 +0000 UTC m=+0.369068691,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-121.ec2.internal,}"
Sep 12 00:43:15.827416 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.827401    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 12 00:43:15.828173 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.828146    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 12 00:43:15.829709 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:15.829265    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-121.ec2.internal\" not found"
Sep 12 00:43:15.829807 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.829730    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 12 00:43:15.829807 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.829744    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 12 00:43:15.829807 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.829805    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 12 00:43:15.829978 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.829810    2567 reconstruct.go:97] "Volume reconstruction finished"
Sep 12 00:43:15.829978 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.829826    2567 reconciler.go:29] "Reconciler: start to sync state"
Sep 12 00:43:15.831124 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.831106    2567 factory.go:55] Registering systemd factory
Sep 12 00:43:15.831124 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.831124    2567 factory.go:223] Registration of the systemd container factory successfully
Sep 12 00:43:15.831353 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.831336    2567 factory.go:153] Registering CRI-O factory
Sep 12 00:43:15.831353 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.831354    2567 factory.go:223] Registration of the crio container factory successfully
Sep 12 00:43:15.831465 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.831402    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 12 00:43:15.831465 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.831420    2567 factory.go:103] Registering Raw factory
Sep 12 00:43:15.831465 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.831436    2567 manager.go:1196] Started watching for new ooms in manager
Sep 12 00:43:15.831733 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.831723    2567 manager.go:349] Starting recovery of all containers
Sep 12 00:43:15.838001 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.837807    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 12 00:43:15.841327 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.841299    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:43:15.843060 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.843038    2567 manager.go:354] Recovery completed
Sep 12 00:43:15.843908 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:15.843841    2567 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-121.ec2.internal\" not found" node="ip-10-0-139-121.ec2.internal"
Sep 12 00:43:15.844310 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:15.844291    2567 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 12 00:43:15.847112 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.847099    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:43:15.851257 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.851243    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:43:15.851335 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.851269    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:43:15.851335 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.851279    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:43:15.851852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.851839    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 12 00:43:15.851916 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.851852    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 12 00:43:15.851916 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.851856    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:43:15.851916 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.851872    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 12 00:43:15.852641 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.852628    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:43:15.852694 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.852653    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:43:15.852694 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.852664    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:43:15.854089 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.854076    2567 policy_none.go:49] "None policy: Start"
Sep 12 00:43:15.854129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.854093    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 12 00:43:15.854129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.854103    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 12 00:43:15.856115 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.856105    2567 policy_none.go:47] "Start"
Sep 12 00:43:15.895527 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.895511    2567 manager.go:337] "Starting Device Plugin manager"
Sep 12 00:43:15.910063 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:15.895549    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 12 00:43:15.910063 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.895562    2567 server.go:87] "Starting device plugin registration server"
Sep 12 00:43:15.910063 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.895890    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 12 00:43:15.910063 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.895904    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 12 00:43:15.910063 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.896043    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 12 00:43:15.910063 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.896133    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 12 00:43:15.910063 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.896142    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 12 00:43:15.910063 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:15.896658    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 12 00:43:15.910063 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:15.896703    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-121.ec2.internal\" not found"
Sep 12 00:43:15.966837 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.966745    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 12 00:43:15.966837 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.966802    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 12 00:43:15.966837 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.966831    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 12 00:43:15.967025 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:15.966877    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 12 00:43:15.969826 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.969808    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:43:15.996010 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.995988    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:43:15.996981 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.996959    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:43:15.997067 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.996998    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:43:15.997067 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.997012    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:43:15.997067 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:15.997041    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.004402 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.004388    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.004457 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.004409    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-121.ec2.internal\": node \"ip-10-0-139-121.ec2.internal\" not found"
Sep 12 00:43:16.067266 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.067228    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-121.ec2.internal"]
Sep 12 00:43:16.067409 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.067319    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:43:16.068755 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.068741    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:43:16.068828 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.068768    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:43:16.068828 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.068778    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:43:16.071152 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.071139    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:43:16.071292 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.071277    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.071350 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.071313    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:43:16.071847 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.071831    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:43:16.071908 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.071859    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:43:16.071908 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.071837    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:43:16.071908 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.071869    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:43:16.071908 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.071888    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:43:16.071908 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.071910    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:43:16.074168 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.074152    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.074247 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.074177    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:43:16.074839 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.074816    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:43:16.074839 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.074837    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:43:16.074963 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.074848    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:43:16.093854 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.093821    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-121.ec2.internal\" not found" node="ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.094007 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.093990    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-121.ec2.internal\" not found"
Sep 12 00:43:16.098317 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.098302    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-121.ec2.internal\" not found" node="ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.194297 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.194265    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-121.ec2.internal\" not found"
Sep 12 00:43:16.230986 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.230875    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9fd7b5f142d9f49960e1ce88e20ea881-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal\" (UID: \"9fd7b5f142d9f49960e1ce88e20ea881\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.230986 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.230940    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fd7b5f142d9f49960e1ce88e20ea881-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal\" (UID: \"9fd7b5f142d9f49960e1ce88e20ea881\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.230986 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.230982    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad9778864c1c592078495eb7196d09e3-config\") pod \"kube-apiserver-proxy-ip-10-0-139-121.ec2.internal\" (UID: \"ad9778864c1c592078495eb7196d09e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.295176 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.295142    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-121.ec2.internal\" not found"
Sep 12 00:43:16.331564 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.331534    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9fd7b5f142d9f49960e1ce88e20ea881-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal\" (UID: \"9fd7b5f142d9f49960e1ce88e20ea881\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.331684 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.331567    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fd7b5f142d9f49960e1ce88e20ea881-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal\" (UID: \"9fd7b5f142d9f49960e1ce88e20ea881\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.331684 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.331591    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad9778864c1c592078495eb7196d09e3-config\") pod \"kube-apiserver-proxy-ip-10-0-139-121.ec2.internal\" (UID: \"ad9778864c1c592078495eb7196d09e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.331684 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.331652    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fd7b5f142d9f49960e1ce88e20ea881-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal\" (UID: \"9fd7b5f142d9f49960e1ce88e20ea881\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.331684 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.331665    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad9778864c1c592078495eb7196d09e3-config\") pod \"kube-apiserver-proxy-ip-10-0-139-121.ec2.internal\" (UID: \"ad9778864c1c592078495eb7196d09e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.331814 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.331654    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9fd7b5f142d9f49960e1ce88e20ea881-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal\" (UID: \"9fd7b5f142d9f49960e1ce88e20ea881\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.395294 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.395258    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-121.ec2.internal\" not found"
Sep 12 00:43:16.397416 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.397398    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.401359 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.401334    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.495874 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.495784    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-121.ec2.internal\" not found"
Sep 12 00:43:16.596291 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.596246    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-121.ec2.internal\" not found"
Sep 12 00:43:16.688710 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.688676    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:43:16.729878 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.729844    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.741548 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.741520    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 12 00:43:16.743221 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.743207    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.746406 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.746349    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 12 00:43:16.746515 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.746496    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 12 00:43:16.746569 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.746517    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 12 00:43:16.746569 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.746515    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 12 00:43:16.746569 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.746553    2567 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a4fdabf2789524bf1b779cbb2abdb105-ce524b211187c4bf.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.139.121:58346->34.231.200.155:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:16.796738 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.796709    2567 apiserver.go:52] "Watching apiserver"
Sep 12 00:43:16.810473 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.810438    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 12 00:43:16.812471 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.812446    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-5k6jv","openshift-ovn-kubernetes/ovnkube-node-srp9g","kube-system/konnectivity-agent-g5brn","openshift-dns/node-resolver-cbjgx","openshift-multus/multus-additional-cni-plugins-qg489","openshift-multus/multus-rnzw4","openshift-multus/network-metrics-daemon-sdhm4","kube-system/global-pull-secret-syncer-5k8lk","kube-system/kube-apiserver-proxy-ip-10-0-139-121.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm","openshift-cluster-node-tuning-operator/tuned-h2vvz","openshift-image-registry/node-ca-mvtjp","openshift-network-diagnostics/network-check-target-bdgdv"]
Sep 12 00:43:16.815592 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.815570    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mvtjp"
Sep 12 00:43:16.818465 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.818132    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 12 00:43:16.818465 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.818289    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 12 00:43:16.818465 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.818426    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nhqpq\""
Sep 12 00:43:16.820238 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.818541    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 12 00:43:16.820238 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.818654    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.820925 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.820907    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g5brn"
Sep 12 00:43:16.821402 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.821386    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 12 00:43:16.821726 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.821711    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 12 00:43:16.821811 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.821743    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 12 00:43:16.822115 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.822101    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 12 00:43:16.822197 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.822144    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xpnns\""
Sep 12 00:43:16.822197 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.822186    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 12 00:43:16.822427 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.822411    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 12 00:43:16.823056 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.823041    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jm9nw\""
Sep 12 00:43:16.823338 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.823314    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 12 00:43:16.823414 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.823391    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 12 00:43:16.823576 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.823562    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5k6jv"
Sep 12 00:43:16.825658 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.825641    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:43:16.825759 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.825697    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r88hb\""
Sep 12 00:43:16.825759 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.825709    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 12 00:43:16.825840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.825829    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 12 00:43:16.826227 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.826205    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.827489 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.827465    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 12 00:43:16.828226 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.828178    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-11 00:38:16 +0000 UTC" deadline="2028-05-09 06:00:09.376266331 +0000 UTC"
Sep 12 00:43:16.828345 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.828224    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14525h16m52.548046377s"
Sep 12 00:43:16.828345 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.828339    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4lghz\""
Sep 12 00:43:16.828457 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.828373    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 12 00:43:16.828504 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.828466    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 12 00:43:16.828773 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.828761    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 12 00:43:16.828919 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.828905    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 12 00:43:16.829029 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.828939    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 12 00:43:16.829204 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.829189    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.831372 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.831352    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rwqgw\""
Sep 12 00:43:16.831739 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.831724    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 12 00:43:16.831995 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.831981    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:16.832071 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.832049    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:16.834209 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834187    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.834291 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834259    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-systemd-units\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.834291 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834285    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-etc-openvswitch\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.834389 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834264    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:16.834389 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834343    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-cni-bin\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.834481 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.834401    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:16.834481 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834408    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-system-cni-dir\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.834481 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834461    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-cni-binary-copy\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.834621 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834488    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.834621 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834513    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9626x\" (UniqueName: \"kubernetes.io/projected/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-kube-api-access-9626x\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.834621 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834536    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-node-log\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.834621 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834562    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/937692a3-80e6-477f-a7f9-7cdbaa4ba628-env-overrides\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.834621 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834584    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh552\" (UniqueName: \"kubernetes.io/projected/937692a3-80e6-477f-a7f9-7cdbaa4ba628-kube-api-access-zh552\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.834797 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834618    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-var-lib-cni-bin\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.834797 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834647    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-run-netns\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.834797 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834661    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c-agent-certs\") pod \"konnectivity-agent-g5brn\" (UID: \"cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c\") " pod="kube-system/konnectivity-agent-g5brn"
Sep 12 00:43:16.834797 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834676    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-multus-cni-dir\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.834797 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834705    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-cnibin\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.834797 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834722    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0f1ca3a-739a-4deb-8f9e-de39d894d430-cni-binary-copy\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.834797 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834735    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bkr9\" (UniqueName: \"kubernetes.io/projected/d0f1ca3a-739a-4deb-8f9e-de39d894d430-kube-api-access-6bkr9\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.834797 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834752    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd6sm\" (UniqueName: \"kubernetes.io/projected/079a162f-f33c-4911-8385-e6d87ccd2231-kube-api-access-nd6sm\") pod \"network-metrics-daemon-sdhm4\" (UID: \"079a162f-f33c-4911-8385-e6d87ccd2231\") " pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:16.835129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834804    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-run-systemd\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.835129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834838    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-run-ovn\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.835129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834865    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-log-socket\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.835129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834891    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqwn5\" (UniqueName: \"kubernetes.io/projected/0f47656b-bd7c-4aa2-9943-3de60c25f96b-kube-api-access-tqwn5\") pod \"iptables-alerter-5k6jv\" (UID: \"0f47656b-bd7c-4aa2-9943-3de60c25f96b\") " pod="openshift-network-operator/iptables-alerter-5k6jv"
Sep 12 00:43:16.835129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834923    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-var-lib-cni-multus\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.835129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.834978    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-run-multus-certs\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.835129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835006    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddxfp\" (UniqueName: \"kubernetes.io/projected/5bf0d680-e19c-4892-8652-58f49ed9d3b3-kube-api-access-ddxfp\") pod \"node-ca-mvtjp\" (UID: \"5bf0d680-e19c-4892-8652-58f49ed9d3b3\") " pod="openshift-image-registry/node-ca-mvtjp"
Sep 12 00:43:16.835129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835020    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.835129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835036    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-cni-netd\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.835129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835051    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/937692a3-80e6-477f-a7f9-7cdbaa4ba628-ovn-node-metrics-cert\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.835129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835064    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-hostroot\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.835129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835080    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d0f1ca3a-739a-4deb-8f9e-de39d894d430-multus-daemon-config\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.835129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835102    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-etc-kubernetes\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.835129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835122    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-kubelet\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835138    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0f47656b-bd7c-4aa2-9943-3de60c25f96b-iptables-alerter-script\") pod \"iptables-alerter-5k6jv\" (UID: \"0f47656b-bd7c-4aa2-9943-3de60c25f96b\") " pod="openshift-network-operator/iptables-alerter-5k6jv"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835152    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-var-lib-kubelet\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835179    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-multus-conf-dir\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835202    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-system-cni-dir\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835222    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-cnibin\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835236    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-slash\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835256    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-run-openvswitch\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835275    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-run-ovn-kubernetes\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835290    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/937692a3-80e6-477f-a7f9-7cdbaa4ba628-ovnkube-config\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835328    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c-konnectivity-ca\") pod \"konnectivity-agent-g5brn\" (UID: \"cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c\") " pod="kube-system/konnectivity-agent-g5brn"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835362    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f47656b-bd7c-4aa2-9943-3de60c25f96b-host-slash\") pod \"iptables-alerter-5k6jv\" (UID: \"0f47656b-bd7c-4aa2-9943-3de60c25f96b\") " pod="openshift-network-operator/iptables-alerter-5k6jv"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5bf0d680-e19c-4892-8652-58f49ed9d3b3-host\") pod \"node-ca-mvtjp\" (UID: \"5bf0d680-e19c-4892-8652-58f49ed9d3b3\") " pod="openshift-image-registry/node-ca-mvtjp"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835410    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs\") pod \"network-metrics-daemon-sdhm4\" (UID: \"079a162f-f33c-4911-8385-e6d87ccd2231\") " pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835439    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-var-lib-openvswitch\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835464    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.835666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835487    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/937692a3-80e6-477f-a7f9-7cdbaa4ba628-ovnkube-script-lib\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.836137 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835510    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-multus-socket-dir-parent\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.836137 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835532    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-run-k8s-cni-cncf-io\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.836137 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835555    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-run-netns\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.836137 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.835578    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5bf0d680-e19c-4892-8652-58f49ed9d3b3-serviceca\") pod \"node-ca-mvtjp\" (UID: \"5bf0d680-e19c-4892-8652-58f49ed9d3b3\") " pod="openshift-image-registry/node-ca-mvtjp"
Sep 12 00:43:16.836678 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.836663    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cbjgx"
Sep 12 00:43:16.838875 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.838857    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:16.838986 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.838970    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gdphh\""
Sep 12 00:43:16.839055 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.839036    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 12 00:43:16.839689 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.839676    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 12 00:43:16.839925 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.839913    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 12 00:43:16.841302 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.841287    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.842731 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.842716    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 12 00:43:16.842835 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.842820    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4z7gv\""
Sep 12 00:43:16.843141 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.843121    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 12 00:43:16.843237 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.843196    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 12 00:43:16.843817 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.843785    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:16.843903 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.843855    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:16.844274 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.844253    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 12 00:43:16.844343 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.844324    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:43:16.844939 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.844921    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8rn8r\""
Sep 12 00:43:16.869939 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.869914    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sz26p"
Sep 12 00:43:16.878201 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.878177    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sz26p"
Sep 12 00:43:16.930564 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.930544    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 12 00:43:16.935746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.935728    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bkr9\" (UniqueName: \"kubernetes.io/projected/d0f1ca3a-739a-4deb-8f9e-de39d894d430-kube-api-access-6bkr9\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.935839 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.935761    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:16.935839 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.935790    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-socket-dir\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:16.935839 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.935839    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-run-netns\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.936130 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.935882    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-cnibin\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.936130 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.935910    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0f1ca3a-739a-4deb-8f9e-de39d894d430-cni-binary-copy\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.936130 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.935991    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-run-netns\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.936130 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.935941    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm8jh\" (UniqueName: \"kubernetes.io/projected/5128643f-5556-48f8-95d9-d82dfa1c3cae-kube-api-access-tm8jh\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:16.936130 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936055    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-cnibin\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.936130 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936061    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2337c617-d3c5-417f-b2c8-4c8328a04a58-hosts-file\") pod \"node-resolver-cbjgx\" (UID: \"2337c617-d3c5-417f-b2c8-4c8328a04a58\") " pod="openshift-dns/node-resolver-cbjgx"
Sep 12 00:43:16.936130 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936106    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-run-systemd\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.936130 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936132    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-run-ovn\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936155    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqwn5\" (UniqueName: \"kubernetes.io/projected/0f47656b-bd7c-4aa2-9943-3de60c25f96b-kube-api-access-tqwn5\") pod \"iptables-alerter-5k6jv\" (UID: \"0f47656b-bd7c-4aa2-9943-3de60c25f96b\") " pod="openshift-network-operator/iptables-alerter-5k6jv"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936159    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-run-systemd\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936194    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-run-ovn\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936218    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-run-multus-certs\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936242    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-sysctl-conf\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936256    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-run-multus-certs\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936259    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-cni-netd\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936288    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-cni-netd\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936292    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/937692a3-80e6-477f-a7f9-7cdbaa4ba628-ovn-node-metrics-cert\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936317    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-kubernetes\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936339    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-sysctl-d\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936359    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-tuned\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936382    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0f47656b-bd7c-4aa2-9943-3de60c25f96b-iptables-alerter-script\") pod \"iptables-alerter-5k6jv\" (UID: \"0f47656b-bd7c-4aa2-9943-3de60c25f96b\") " pod="openshift-network-operator/iptables-alerter-5k6jv"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936412    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-var-lib-kubelet\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936433    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-multus-conf-dir\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936460    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-sysconfig\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.936567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936482    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86crm\" (UniqueName: \"kubernetes.io/projected/2337c617-d3c5-417f-b2c8-4c8328a04a58-kube-api-access-86crm\") pod \"node-resolver-cbjgx\" (UID: \"2337c617-d3c5-417f-b2c8-4c8328a04a58\") " pod="openshift-dns/node-resolver-cbjgx"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936507    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-var-lib-cni-bin\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936527    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-cnibin\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936551    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-slash\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936575    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-run-openvswitch\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936599    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/937692a3-80e6-477f-a7f9-7cdbaa4ba628-ovnkube-config\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936602    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-var-lib-kubelet\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936618    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-run-k8s-cni-cncf-io\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936638    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-run-netns\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936551    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0f1ca3a-739a-4deb-8f9e-de39d894d430-cni-binary-copy\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936705    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-run-netns\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936706    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-cnibin\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936733    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-var-lib-cni-bin\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936734    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-device-dir\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936744    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-slash\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936750    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936763    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5bf0d680-e19c-4892-8652-58f49ed9d3b3-host\") pod \"node-ca-mvtjp\" (UID: \"5bf0d680-e19c-4892-8652-58f49ed9d3b3\") " pod="openshift-image-registry/node-ca-mvtjp"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936774    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-run-openvswitch\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.937534 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936778    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-multus-conf-dir\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936779    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs\") pod \"network-metrics-daemon-sdhm4\" (UID: \"079a162f-f33c-4911-8385-e6d87ccd2231\") " pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936812    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5bf0d680-e19c-4892-8652-58f49ed9d3b3-host\") pod \"node-ca-mvtjp\" (UID: \"5bf0d680-e19c-4892-8652-58f49ed9d3b3\") " pod="openshift-image-registry/node-ca-mvtjp"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936814    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-var-lib-openvswitch\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936841    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936871    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-run-k8s-cni-cncf-io\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936905    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0f47656b-bd7c-4aa2-9943-3de60c25f96b-iptables-alerter-script\") pod \"iptables-alerter-5k6jv\" (UID: \"0f47656b-bd7c-4aa2-9943-3de60c25f96b\") " pod="openshift-network-operator/iptables-alerter-5k6jv"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936920    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-systemd\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936990    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-sys\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.936995    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-var-lib-openvswitch\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937022    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-var-lib-kubelet\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.937034    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937036    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937043    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xmmg\" (UniqueName: \"kubernetes.io/projected/1bb2ff8c-97ad-4543-975f-caed929a31c8-kube-api-access-6xmmg\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:16.937116    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs podName:079a162f-f33c-4911-8385-e6d87ccd2231 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:17.437089662 +0000 UTC m=+1.997541308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs") pod "network-metrics-daemon-sdhm4" (UID: "079a162f-f33c-4911-8385-e6d87ccd2231") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937133    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5bf0d680-e19c-4892-8652-58f49ed9d3b3-serviceca\") pod \"node-ca-mvtjp\" (UID: \"5bf0d680-e19c-4892-8652-58f49ed9d3b3\") " pod="openshift-image-registry/node-ca-mvtjp"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937173    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.938330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937260    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/937692a3-80e6-477f-a7f9-7cdbaa4ba628-ovnkube-config\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937272    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-systemd-units\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937326    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-systemd-units\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937337    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-etc-openvswitch\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937366    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-system-cni-dir\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937388    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-dbus\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937394    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937407    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9626x\" (UniqueName: \"kubernetes.io/projected/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-kube-api-access-9626x\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937426    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-etc-openvswitch\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937483    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-system-cni-dir\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937495    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-node-log\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937578    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-sys-fs\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937576    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5bf0d680-e19c-4892-8652-58f49ed9d3b3-serviceca\") pod \"node-ca-mvtjp\" (UID: \"5bf0d680-e19c-4892-8652-58f49ed9d3b3\") " pod="openshift-image-registry/node-ca-mvtjp"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937601    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-node-log\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937636    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c-agent-certs\") pod \"konnectivity-agent-g5brn\" (UID: \"cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c\") " pod="kube-system/konnectivity-agent-g5brn"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937658    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-multus-cni-dir\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937699    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-registration-dir\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:16.939185 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937709    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-multus-cni-dir\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937733    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-etc-selinux\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937754    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-lib-modules\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937773    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2337c617-d3c5-417f-b2c8-4c8328a04a58-tmp-dir\") pod \"node-resolver-cbjgx\" (UID: \"2337c617-d3c5-417f-b2c8-4c8328a04a58\") " pod="openshift-dns/node-resolver-cbjgx"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937797    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd6sm\" (UniqueName: \"kubernetes.io/projected/079a162f-f33c-4911-8385-e6d87ccd2231-kube-api-access-nd6sm\") pod \"network-metrics-daemon-sdhm4\" (UID: \"079a162f-f33c-4911-8385-e6d87ccd2231\") " pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937820    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-log-socket\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937846    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-var-lib-cni-multus\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937870    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-run\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937895    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddxfp\" (UniqueName: \"kubernetes.io/projected/5bf0d680-e19c-4892-8652-58f49ed9d3b3-kube-api-access-ddxfp\") pod \"node-ca-mvtjp\" (UID: \"5bf0d680-e19c-4892-8652-58f49ed9d3b3\") " pod="openshift-image-registry/node-ca-mvtjp"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937905    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-log-socket\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937920    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937966    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d0f1ca3a-739a-4deb-8f9e-de39d894d430-multus-daemon-config\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.937992    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-etc-kubernetes\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938017    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-host\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938043    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb2ff8c-97ad-4543-975f-caed929a31c8-tmp\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938066    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938089    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-kubelet\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.939746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938113    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-hostroot\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938151    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-host-var-lib-cni-multus\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938166    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-system-cni-dir\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938194    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-run-ovn-kubernetes\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938233    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c-konnectivity-ca\") pod \"konnectivity-agent-g5brn\" (UID: \"cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c\") " pod="kube-system/konnectivity-agent-g5brn"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938258    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f47656b-bd7c-4aa2-9943-3de60c25f96b-host-slash\") pod \"iptables-alerter-5k6jv\" (UID: \"0f47656b-bd7c-4aa2-9943-3de60c25f96b\") " pod="openshift-network-operator/iptables-alerter-5k6jv"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938281    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-modprobe-d\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938305    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlkbs\" (UniqueName: \"kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs\") pod \"network-check-target-bdgdv\" (UID: \"32f044d9-76e0-48fd-bd47-9f829d5a74d5\") " pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938330    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/937692a3-80e6-477f-a7f9-7cdbaa4ba628-ovnkube-script-lib\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938357    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-multus-socket-dir-parent\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938407    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-run-ovn-kubernetes\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938418    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-multus-socket-dir-parent\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938645    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-kubelet\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938686    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-hostroot\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938725    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-system-cni-dir\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938749    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-kubelet-config\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938780    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-cni-bin\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.940289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938804    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-cni-binary-copy\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.940792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938829    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.940792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938841    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.940792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938850    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c-konnectivity-ca\") pod \"konnectivity-agent-g5brn\" (UID: \"cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c\") " pod="kube-system/konnectivity-agent-g5brn"
Sep 12 00:43:16.940792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938854    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/937692a3-80e6-477f-a7f9-7cdbaa4ba628-env-overrides\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.940792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.938934    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh552\" (UniqueName: \"kubernetes.io/projected/937692a3-80e6-477f-a7f9-7cdbaa4ba628-kube-api-access-zh552\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.940792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.939002    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f47656b-bd7c-4aa2-9943-3de60c25f96b-host-slash\") pod \"iptables-alerter-5k6jv\" (UID: \"0f47656b-bd7c-4aa2-9943-3de60c25f96b\") " pod="openshift-network-operator/iptables-alerter-5k6jv"
Sep 12 00:43:16.940792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.939060    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0f1ca3a-739a-4deb-8f9e-de39d894d430-etc-kubernetes\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.940792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.939098    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/937692a3-80e6-477f-a7f9-7cdbaa4ba628-host-cni-bin\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.940792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.939234    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d0f1ca3a-739a-4deb-8f9e-de39d894d430-multus-daemon-config\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.940792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.939296    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/937692a3-80e6-477f-a7f9-7cdbaa4ba628-env-overrides\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.940792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.939519    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/937692a3-80e6-477f-a7f9-7cdbaa4ba628-ovnkube-script-lib\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.940792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.939577    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.940792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.939861    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-cni-binary-copy\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.940792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.940711    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/937692a3-80e6-477f-a7f9-7cdbaa4ba628-ovn-node-metrics-cert\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.941233 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.940866    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c-agent-certs\") pod \"konnectivity-agent-g5brn\" (UID: \"cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c\") " pod="kube-system/konnectivity-agent-g5brn"
Sep 12 00:43:16.946963 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.946901    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqwn5\" (UniqueName: \"kubernetes.io/projected/0f47656b-bd7c-4aa2-9943-3de60c25f96b-kube-api-access-tqwn5\") pod \"iptables-alerter-5k6jv\" (UID: \"0f47656b-bd7c-4aa2-9943-3de60c25f96b\") " pod="openshift-network-operator/iptables-alerter-5k6jv"
Sep 12 00:43:16.947461 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.947442    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bkr9\" (UniqueName: \"kubernetes.io/projected/d0f1ca3a-739a-4deb-8f9e-de39d894d430-kube-api-access-6bkr9\") pod \"multus-rnzw4\" (UID: \"d0f1ca3a-739a-4deb-8f9e-de39d894d430\") " pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:16.947528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.947446    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9626x\" (UniqueName: \"kubernetes.io/projected/f2ef7878-3d24-42f5-9708-2fe05bbe4bdf-kube-api-access-9626x\") pod \"multus-additional-cni-plugins-qg489\" (UID: \"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf\") " pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:16.948417 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.948403    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddxfp\" (UniqueName: \"kubernetes.io/projected/5bf0d680-e19c-4892-8652-58f49ed9d3b3-kube-api-access-ddxfp\") pod \"node-ca-mvtjp\" (UID: \"5bf0d680-e19c-4892-8652-58f49ed9d3b3\") " pod="openshift-image-registry/node-ca-mvtjp"
Sep 12 00:43:16.949232 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.949209    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd6sm\" (UniqueName: \"kubernetes.io/projected/079a162f-f33c-4911-8385-e6d87ccd2231-kube-api-access-nd6sm\") pod \"network-metrics-daemon-sdhm4\" (UID: \"079a162f-f33c-4911-8385-e6d87ccd2231\") " pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:16.949232 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.949222    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh552\" (UniqueName: \"kubernetes.io/projected/937692a3-80e6-477f-a7f9-7cdbaa4ba628-kube-api-access-zh552\") pod \"ovnkube-node-srp9g\" (UID: \"937692a3-80e6-477f-a7f9-7cdbaa4ba628\") " pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:16.990348 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:16.990320    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9778864c1c592078495eb7196d09e3.slice/crio-e94a537a5c64a96e55111c4a208d7115aa254922a54387576a7dfa18343c7e31 WatchSource:0}: Error finding container e94a537a5c64a96e55111c4a208d7115aa254922a54387576a7dfa18343c7e31: Status 404 returned error can't find the container with id e94a537a5c64a96e55111c4a208d7115aa254922a54387576a7dfa18343c7e31
Sep 12 00:43:16.990601 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:16.990576    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd7b5f142d9f49960e1ce88e20ea881.slice/crio-c25e5e5cd00b9cb0d02b23722774aa073cd51376103541f2c464cf4d157269fa WatchSource:0}: Error finding container c25e5e5cd00b9cb0d02b23722774aa073cd51376103541f2c464cf4d157269fa: Status 404 returned error can't find the container with id c25e5e5cd00b9cb0d02b23722774aa073cd51376103541f2c464cf4d157269fa
Sep 12 00:43:16.995513 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:16.995494    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:43:17.039316 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039286    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-dbus\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:17.039316 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039318    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-sys-fs\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039335    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-registration-dir\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039352    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-etc-selinux\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039373    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-lib-modules\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039394    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2337c617-d3c5-417f-b2c8-4c8328a04a58-tmp-dir\") pod \"node-resolver-cbjgx\" (UID: \"2337c617-d3c5-417f-b2c8-4c8328a04a58\") " pod="openshift-dns/node-resolver-cbjgx"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-run\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039423    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-sys-fs\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039442    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-host\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039461    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb2ff8c-97ad-4543-975f-caed929a31c8-tmp\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039484    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-run\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039487    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039491    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-registration-dir\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039500    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-dbus\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039492    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-etc-selinux\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039522    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-modprobe-d\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.039528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039527    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-host\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039534    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-lib-modules\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039597    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-modprobe-d\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:17.039601    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039635    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlkbs\" (UniqueName: \"kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs\") pod \"network-check-target-bdgdv\" (UID: \"32f044d9-76e0-48fd-bd47-9f829d5a74d5\") " pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:17.039669    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret podName:d5d08bf0-f998-4798-b16c-ec2c5cecbe21 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:17.539648944 +0000 UTC m=+2.100100604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret") pod "global-pull-secret-syncer-5k8lk" (UID: "d5d08bf0-f998-4798-b16c-ec2c5cecbe21") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039703    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-kubelet-config\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039738    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039750    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2337c617-d3c5-417f-b2c8-4c8328a04a58-tmp-dir\") pod \"node-resolver-cbjgx\" (UID: \"2337c617-d3c5-417f-b2c8-4c8328a04a58\") " pod="openshift-dns/node-resolver-cbjgx"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039759    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-kubelet-config\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039761    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-socket-dir\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039803    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039818    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm8jh\" (UniqueName: \"kubernetes.io/projected/5128643f-5556-48f8-95d9-d82dfa1c3cae-kube-api-access-tm8jh\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039843    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2337c617-d3c5-417f-b2c8-4c8328a04a58-hosts-file\") pod \"node-resolver-cbjgx\" (UID: \"2337c617-d3c5-417f-b2c8-4c8328a04a58\") " pod="openshift-dns/node-resolver-cbjgx"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039853    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-socket-dir\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039972    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-sysctl-conf\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.039995    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-kubernetes\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.040253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040006    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2337c617-d3c5-417f-b2c8-4c8328a04a58-hosts-file\") pod \"node-resolver-cbjgx\" (UID: \"2337c617-d3c5-417f-b2c8-4c8328a04a58\") " pod="openshift-dns/node-resolver-cbjgx"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040011    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-sysctl-d\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040058    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-tuned\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040079    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-sysconfig\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040106    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-kubernetes\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040082    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-sysctl-d\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040109    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86crm\" (UniqueName: \"kubernetes.io/projected/2337c617-d3c5-417f-b2c8-4c8328a04a58-kube-api-access-86crm\") pod \"node-resolver-cbjgx\" (UID: \"2337c617-d3c5-417f-b2c8-4c8328a04a58\") " pod="openshift-dns/node-resolver-cbjgx"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040148    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-sysctl-conf\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040158    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-sysconfig\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040163    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-device-dir\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040209    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5128643f-5556-48f8-95d9-d82dfa1c3cae-device-dir\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040208    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-systemd\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040247    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-systemd\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040248    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-sys\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040282    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-sys\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040282    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-var-lib-kubelet\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040319    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xmmg\" (UniqueName: \"kubernetes.io/projected/1bb2ff8c-97ad-4543-975f-caed929a31c8-kube-api-access-6xmmg\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.041296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.040319    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb2ff8c-97ad-4543-975f-caed929a31c8-var-lib-kubelet\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.042259 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.042242    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1bb2ff8c-97ad-4543-975f-caed929a31c8-etc-tuned\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.042480 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.042463    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb2ff8c-97ad-4543-975f-caed929a31c8-tmp\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.046317 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:17.046303    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:43:17.046373 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:17.046321    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:43:17.046373 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:17.046331    2567 projected.go:196] Error preparing data for projected volume kube-api-access-hlkbs for pod openshift-network-diagnostics/network-check-target-bdgdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:17.046475 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:17.046383    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs podName:32f044d9-76e0-48fd-bd47-9f829d5a74d5 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:17.546371596 +0000 UTC m=+2.106823258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hlkbs" (UniqueName: "kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs") pod "network-check-target-bdgdv" (UID: "32f044d9-76e0-48fd-bd47-9f829d5a74d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:17.048536 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.048514    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm8jh\" (UniqueName: \"kubernetes.io/projected/5128643f-5556-48f8-95d9-d82dfa1c3cae-kube-api-access-tm8jh\") pod \"aws-ebs-csi-driver-node-6l4zm\" (UID: \"5128643f-5556-48f8-95d9-d82dfa1c3cae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.048731 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.048715    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86crm\" (UniqueName: \"kubernetes.io/projected/2337c617-d3c5-417f-b2c8-4c8328a04a58-kube-api-access-86crm\") pod \"node-resolver-cbjgx\" (UID: \"2337c617-d3c5-417f-b2c8-4c8328a04a58\") " pod="openshift-dns/node-resolver-cbjgx"
Sep 12 00:43:17.048772 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.048713    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xmmg\" (UniqueName: \"kubernetes.io/projected/1bb2ff8c-97ad-4543-975f-caed929a31c8-kube-api-access-6xmmg\") pod \"tuned-h2vvz\" (UID: \"1bb2ff8c-97ad-4543-975f-caed929a31c8\") " pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.144331 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.144299    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mvtjp"
Sep 12 00:43:17.154110 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:17.154081    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf0d680_e19c_4892_8652_58f49ed9d3b3.slice/crio-7e63807ee38664aa0301aa88b2983bdcd38f91629fc460d59e9c846ca05f6fb0 WatchSource:0}: Error finding container 7e63807ee38664aa0301aa88b2983bdcd38f91629fc460d59e9c846ca05f6fb0: Status 404 returned error can't find the container with id 7e63807ee38664aa0301aa88b2983bdcd38f91629fc460d59e9c846ca05f6fb0
Sep 12 00:43:17.162275 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.162249    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:17.170570 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:17.170544    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937692a3_80e6_477f_a7f9_7cdbaa4ba628.slice/crio-0fb72c939d1e94767c39fdefc9a8f45e763c0a9d31d925534ac7e99a760d1a96 WatchSource:0}: Error finding container 0fb72c939d1e94767c39fdefc9a8f45e763c0a9d31d925534ac7e99a760d1a96: Status 404 returned error can't find the container with id 0fb72c939d1e94767c39fdefc9a8f45e763c0a9d31d925534ac7e99a760d1a96
Sep 12 00:43:17.177176 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.177158    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g5brn"
Sep 12 00:43:17.185111 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:17.185088    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7cbf7f_c916_4a10_a6f2_4e562a1edc5c.slice/crio-a27abda990b04777fbfa32b9117eceaecb38572823f24aeb57aa92316e013197 WatchSource:0}: Error finding container a27abda990b04777fbfa32b9117eceaecb38572823f24aeb57aa92316e013197: Status 404 returned error can't find the container with id a27abda990b04777fbfa32b9117eceaecb38572823f24aeb57aa92316e013197
Sep 12 00:43:17.192095 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.191914    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5k6jv"
Sep 12 00:43:17.198144 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.198126    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qg489"
Sep 12 00:43:17.201070 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:17.201032    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f47656b_bd7c_4aa2_9943_3de60c25f96b.slice/crio-e839fa9883c1ac4f82e619f5b29b4845e941780f887d6a75206519ac3ec915c5 WatchSource:0}: Error finding container e839fa9883c1ac4f82e619f5b29b4845e941780f887d6a75206519ac3ec915c5: Status 404 returned error can't find the container with id e839fa9883c1ac4f82e619f5b29b4845e941780f887d6a75206519ac3ec915c5
Sep 12 00:43:17.207598 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:17.207578    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ef7878_3d24_42f5_9708_2fe05bbe4bdf.slice/crio-27ffafd2b5d710a8b6c2335816e303be7f2bf7e6942a2914662c5c18901ad393 WatchSource:0}: Error finding container 27ffafd2b5d710a8b6c2335816e303be7f2bf7e6942a2914662c5c18901ad393: Status 404 returned error can't find the container with id 27ffafd2b5d710a8b6c2335816e303be7f2bf7e6942a2914662c5c18901ad393
Sep 12 00:43:17.211914 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.211896    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rnzw4"
Sep 12 00:43:17.217985 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.217967    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cbjgx"
Sep 12 00:43:17.222185 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:17.222160    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f1ca3a_739a_4deb_8f9e_de39d894d430.slice/crio-4712349832a9408bd6943bd1c956797dd7c013af5ee92aa8dc8c060e3d5ba16a WatchSource:0}: Error finding container 4712349832a9408bd6943bd1c956797dd7c013af5ee92aa8dc8c060e3d5ba16a: Status 404 returned error can't find the container with id 4712349832a9408bd6943bd1c956797dd7c013af5ee92aa8dc8c060e3d5ba16a
Sep 12 00:43:17.227196 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:17.227177    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2337c617_d3c5_417f_b2c8_4c8328a04a58.slice/crio-7a20a6e390a612fd00ae31ad97c2d9d12f059dda4c89ef0dc6a54fe6f60ad08b WatchSource:0}: Error finding container 7a20a6e390a612fd00ae31ad97c2d9d12f059dda4c89ef0dc6a54fe6f60ad08b: Status 404 returned error can't find the container with id 7a20a6e390a612fd00ae31ad97c2d9d12f059dda4c89ef0dc6a54fe6f60ad08b
Sep 12 00:43:17.233619 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.233599    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm"
Sep 12 00:43:17.240306 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.240273    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h2vvz"
Sep 12 00:43:17.245013 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:17.244937    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5128643f_5556_48f8_95d9_d82dfa1c3cae.slice/crio-6fa9c2c83c1af882bafcf2c41df2902545c5afff1d60a7734ce9c3bbb9e4ac04 WatchSource:0}: Error finding container 6fa9c2c83c1af882bafcf2c41df2902545c5afff1d60a7734ce9c3bbb9e4ac04: Status 404 returned error can't find the container with id 6fa9c2c83c1af882bafcf2c41df2902545c5afff1d60a7734ce9c3bbb9e4ac04
Sep 12 00:43:17.250382 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:17.250360    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb2ff8c_97ad_4543_975f_caed929a31c8.slice/crio-ddae5c7291a661ff1684892e962a6d19bb6fc7e09d1d4fbf5ef31f54409230c5 WatchSource:0}: Error finding container ddae5c7291a661ff1684892e962a6d19bb6fc7e09d1d4fbf5ef31f54409230c5: Status 404 returned error can't find the container with id ddae5c7291a661ff1684892e962a6d19bb6fc7e09d1d4fbf5ef31f54409230c5
Sep 12 00:43:17.385091 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.385061    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:43:17.443965 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.443917    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs\") pod \"network-metrics-daemon-sdhm4\" (UID: \"079a162f-f33c-4911-8385-e6d87ccd2231\") " pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:17.444116 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:17.444073    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:17.444164 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:17.444144    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs podName:079a162f-f33c-4911-8385-e6d87ccd2231 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:18.444128193 +0000 UTC m=+3.004579842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs") pod "network-metrics-daemon-sdhm4" (UID: "079a162f-f33c-4911-8385-e6d87ccd2231") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:17.544647 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.544483    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:17.544822 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:17.544660    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:17.544822 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:17.544735    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret podName:d5d08bf0-f998-4798-b16c-ec2c5cecbe21 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:18.544719212 +0000 UTC m=+3.105170878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret") pod "global-pull-secret-syncer-5k8lk" (UID: "d5d08bf0-f998-4798-b16c-ec2c5cecbe21") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:17.645713 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.645670    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlkbs\" (UniqueName: \"kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs\") pod \"network-check-target-bdgdv\" (UID: \"32f044d9-76e0-48fd-bd47-9f829d5a74d5\") " pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:17.645940 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:17.645898    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:43:17.645940 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:17.645927    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:43:17.645940 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:17.645939    2567 projected.go:196] Error preparing data for projected volume kube-api-access-hlkbs for pod openshift-network-diagnostics/network-check-target-bdgdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:17.646246 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:17.646043    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs podName:32f044d9-76e0-48fd-bd47-9f829d5a74d5 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:18.645985691 +0000 UTC m=+3.206437341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hlkbs" (UniqueName: "kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs") pod "network-check-target-bdgdv" (UID: "32f044d9-76e0-48fd-bd47-9f829d5a74d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:17.789090 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.788230    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:43:17.881387 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.881201    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 00:38:17 +0000 UTC" deadline="2028-06-07 04:08:42.897572936 +0000 UTC"
Sep 12 00:43:17.881387 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.881262    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15219h25m25.016314691s"
Sep 12 00:43:17.994397 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:17.994255    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h2vvz" event={"ID":"1bb2ff8c-97ad-4543-975f-caed929a31c8","Type":"ContainerStarted","Data":"ddae5c7291a661ff1684892e962a6d19bb6fc7e09d1d4fbf5ef31f54409230c5"}
Sep 12 00:43:18.022281 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.022244    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cbjgx" event={"ID":"2337c617-d3c5-417f-b2c8-4c8328a04a58","Type":"ContainerStarted","Data":"7a20a6e390a612fd00ae31ad97c2d9d12f059dda4c89ef0dc6a54fe6f60ad08b"}
Sep 12 00:43:18.037889 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.037856    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5k6jv" event={"ID":"0f47656b-bd7c-4aa2-9943-3de60c25f96b","Type":"ContainerStarted","Data":"e839fa9883c1ac4f82e619f5b29b4845e941780f887d6a75206519ac3ec915c5"}
Sep 12 00:43:18.057766 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.057527    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g5brn" event={"ID":"cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c","Type":"ContainerStarted","Data":"a27abda990b04777fbfa32b9117eceaecb38572823f24aeb57aa92316e013197"}
Sep 12 00:43:18.063253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.063217    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g" event={"ID":"937692a3-80e6-477f-a7f9-7cdbaa4ba628","Type":"ContainerStarted","Data":"0fb72c939d1e94767c39fdefc9a8f45e763c0a9d31d925534ac7e99a760d1a96"}
Sep 12 00:43:18.080217 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.080176    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-121.ec2.internal" event={"ID":"ad9778864c1c592078495eb7196d09e3","Type":"ContainerStarted","Data":"e94a537a5c64a96e55111c4a208d7115aa254922a54387576a7dfa18343c7e31"}
Sep 12 00:43:18.101302 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.101272    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:43:18.101862 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.101832    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm" event={"ID":"5128643f-5556-48f8-95d9-d82dfa1c3cae","Type":"ContainerStarted","Data":"6fa9c2c83c1af882bafcf2c41df2902545c5afff1d60a7734ce9c3bbb9e4ac04"}
Sep 12 00:43:18.120824 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.120788    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rnzw4" event={"ID":"d0f1ca3a-739a-4deb-8f9e-de39d894d430","Type":"ContainerStarted","Data":"4712349832a9408bd6943bd1c956797dd7c013af5ee92aa8dc8c060e3d5ba16a"}
Sep 12 00:43:18.135840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.135687    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg489" event={"ID":"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf","Type":"ContainerStarted","Data":"27ffafd2b5d710a8b6c2335816e303be7f2bf7e6942a2914662c5c18901ad393"}
Sep 12 00:43:18.151249 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.151210    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mvtjp" event={"ID":"5bf0d680-e19c-4892-8652-58f49ed9d3b3","Type":"ContainerStarted","Data":"7e63807ee38664aa0301aa88b2983bdcd38f91629fc460d59e9c846ca05f6fb0"}
Sep 12 00:43:18.161245 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.161208    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal" event={"ID":"9fd7b5f142d9f49960e1ce88e20ea881","Type":"ContainerStarted","Data":"c25e5e5cd00b9cb0d02b23722774aa073cd51376103541f2c464cf4d157269fa"}
Sep 12 00:43:18.454246 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.453502    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs\") pod \"network-metrics-daemon-sdhm4\" (UID: \"079a162f-f33c-4911-8385-e6d87ccd2231\") " pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:18.454246 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:18.453720    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:18.454246 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:18.453789    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs podName:079a162f-f33c-4911-8385-e6d87ccd2231 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:20.453768055 +0000 UTC m=+5.014219723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs") pod "network-metrics-daemon-sdhm4" (UID: "079a162f-f33c-4911-8385-e6d87ccd2231") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:18.554822 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.554239    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:18.554822 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:18.554405    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:18.554822 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:18.554478    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret podName:d5d08bf0-f998-4798-b16c-ec2c5cecbe21 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:20.554459776 +0000 UTC m=+5.114911430 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret") pod "global-pull-secret-syncer-5k8lk" (UID: "d5d08bf0-f998-4798-b16c-ec2c5cecbe21") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:18.655373 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.654601    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlkbs\" (UniqueName: \"kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs\") pod \"network-check-target-bdgdv\" (UID: \"32f044d9-76e0-48fd-bd47-9f829d5a74d5\") " pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:18.655373 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:18.654861    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:43:18.655373 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:18.654882    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:43:18.655373 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:18.654896    2567 projected.go:196] Error preparing data for projected volume kube-api-access-hlkbs for pod openshift-network-diagnostics/network-check-target-bdgdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:18.655373 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:18.654960    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs podName:32f044d9-76e0-48fd-bd47-9f829d5a74d5 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:20.65494144 +0000 UTC m=+5.215393090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hlkbs" (UniqueName: "kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs") pod "network-check-target-bdgdv" (UID: "32f044d9-76e0-48fd-bd47-9f829d5a74d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:18.660818 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.660782    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:43:18.882502 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.882405    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 00:38:17 +0000 UTC" deadline="2028-02-09 15:27:04.056904737 +0000 UTC"
Sep 12 00:43:18.882502 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.882452    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12374h43m45.174458132s"
Sep 12 00:43:18.967190 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.967145    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:18.967385 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:18.967301    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:18.967775 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.967755    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:18.967863 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:18.967850    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:18.967927 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:18.967915    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:18.968035 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:18.967985    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:20.469352 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:20.469309    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs\") pod \"network-metrics-daemon-sdhm4\" (UID: \"079a162f-f33c-4911-8385-e6d87ccd2231\") " pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:20.469964 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:20.469515    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:20.469964 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:20.469588    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs podName:079a162f-f33c-4911-8385-e6d87ccd2231 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:24.469566056 +0000 UTC m=+9.030017719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs") pod "network-metrics-daemon-sdhm4" (UID: "079a162f-f33c-4911-8385-e6d87ccd2231") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:20.570738 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:20.570695    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:20.570928 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:20.570907    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:20.570988 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:20.570974    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret podName:d5d08bf0-f998-4798-b16c-ec2c5cecbe21 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:24.57095565 +0000 UTC m=+9.131407304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret") pod "global-pull-secret-syncer-5k8lk" (UID: "d5d08bf0-f998-4798-b16c-ec2c5cecbe21") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:20.671755 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:20.671712    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlkbs\" (UniqueName: \"kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs\") pod \"network-check-target-bdgdv\" (UID: \"32f044d9-76e0-48fd-bd47-9f829d5a74d5\") " pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:20.671952 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:20.671932    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:43:20.672046 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:20.671960    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:43:20.672046 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:20.671999    2567 projected.go:196] Error preparing data for projected volume kube-api-access-hlkbs for pod openshift-network-diagnostics/network-check-target-bdgdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:20.672225 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:20.672065    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs podName:32f044d9-76e0-48fd-bd47-9f829d5a74d5 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:24.67204473 +0000 UTC m=+9.232496398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hlkbs" (UniqueName: "kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs") pod "network-check-target-bdgdv" (UID: "32f044d9-76e0-48fd-bd47-9f829d5a74d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:20.968169 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:20.967667    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:20.968169 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:20.967808    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:20.968169 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:20.967872    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:20.968169 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:20.967958    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:20.968169 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:20.967999    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:20.968169 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:20.968086    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:22.967599 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:22.967558    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:22.968142 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:22.967721    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:22.968142 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:22.968116    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:22.968259 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:22.968209    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:22.968322 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:22.968264    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:22.968374 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:22.968345    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:24.506691 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:24.506650    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs\") pod \"network-metrics-daemon-sdhm4\" (UID: \"079a162f-f33c-4911-8385-e6d87ccd2231\") " pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:24.507141 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:24.506807    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:24.507141 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:24.506877    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs podName:079a162f-f33c-4911-8385-e6d87ccd2231 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:32.506853953 +0000 UTC m=+17.067305608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs") pod "network-metrics-daemon-sdhm4" (UID: "079a162f-f33c-4911-8385-e6d87ccd2231") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:24.607090 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:24.607047    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:24.607282 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:24.607222    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:24.607337 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:24.607286    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret podName:d5d08bf0-f998-4798-b16c-ec2c5cecbe21 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:32.607265167 +0000 UTC m=+17.167716820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret") pod "global-pull-secret-syncer-5k8lk" (UID: "d5d08bf0-f998-4798-b16c-ec2c5cecbe21") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:24.707742 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:24.707654    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlkbs\" (UniqueName: \"kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs\") pod \"network-check-target-bdgdv\" (UID: \"32f044d9-76e0-48fd-bd47-9f829d5a74d5\") " pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:24.707920 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:24.707868    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:43:24.707920 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:24.707896    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:43:24.707920 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:24.707909    2567 projected.go:196] Error preparing data for projected volume kube-api-access-hlkbs for pod openshift-network-diagnostics/network-check-target-bdgdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:24.708109 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:24.708001    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs podName:32f044d9-76e0-48fd-bd47-9f829d5a74d5 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:32.707977981 +0000 UTC m=+17.268429643 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hlkbs" (UniqueName: "kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs") pod "network-check-target-bdgdv" (UID: "32f044d9-76e0-48fd-bd47-9f829d5a74d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:24.968244 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:24.968152    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:24.968244 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:24.968189    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:24.968472 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:24.968157    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:24.968472 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:24.968293    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:24.968472 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:24.968412    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:24.968627 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:24.968492    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:26.967681 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:26.967648    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:26.968144 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:26.967691    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:26.968144 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:26.967762    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:26.968144 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:26.967775    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:26.968144 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:26.967875    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:26.968144 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:26.967966    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:28.967026 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:28.966994    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:28.967451 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:28.966994    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:28.967451 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:28.967116    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:28.967451 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:28.966998    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:28.967451 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:28.967211    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:28.967451 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:28.967293    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:30.967378 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:30.967339    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:30.967981 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:30.967340    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:30.967981 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:30.967496    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:30.967981 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:30.967494    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:30.967981 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:30.967599    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:30.967981 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:30.967659    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:32.570095 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:32.570059    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs\") pod \"network-metrics-daemon-sdhm4\" (UID: \"079a162f-f33c-4911-8385-e6d87ccd2231\") " pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:32.570591 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:32.570227    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:32.570591 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:32.570304    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs podName:079a162f-f33c-4911-8385-e6d87ccd2231 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:48.570283384 +0000 UTC m=+33.130735031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs") pod "network-metrics-daemon-sdhm4" (UID: "079a162f-f33c-4911-8385-e6d87ccd2231") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:32.671422 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:32.671378    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:32.671610 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:32.671481    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:32.671610 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:32.671545    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret podName:d5d08bf0-f998-4798-b16c-ec2c5cecbe21 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:48.671528069 +0000 UTC m=+33.231979718 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret") pod "global-pull-secret-syncer-5k8lk" (UID: "d5d08bf0-f998-4798-b16c-ec2c5cecbe21") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:32.772781 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:32.772743    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlkbs\" (UniqueName: \"kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs\") pod \"network-check-target-bdgdv\" (UID: \"32f044d9-76e0-48fd-bd47-9f829d5a74d5\") " pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:32.772968 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:32.772907    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:43:32.772968 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:32.772927    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:43:32.772968 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:32.772939    2567 projected.go:196] Error preparing data for projected volume kube-api-access-hlkbs for pod openshift-network-diagnostics/network-check-target-bdgdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:32.773123 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:32.773022    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs podName:32f044d9-76e0-48fd-bd47-9f829d5a74d5 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:48.773005985 +0000 UTC m=+33.333457637 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hlkbs" (UniqueName: "kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs") pod "network-check-target-bdgdv" (UID: "32f044d9-76e0-48fd-bd47-9f829d5a74d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:32.967244 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:32.967167    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:32.967244 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:32.967182    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:32.967244 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:32.967182    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:32.967563 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:32.967278    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:32.967563 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:32.967338    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:32.967563 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:32.967406    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:34.967470 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:34.967433    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:34.968072 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:34.967433    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:34.968072 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:34.967571    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:34.968072 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:34.967433    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:34.968072 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:34.967619    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:34.968072 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:34.967712    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:36.967056 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:36.967022    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:36.967056 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:36.967050    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:36.967482 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:36.967029    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:36.967482 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:36.967144    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:36.967482 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:36.967229    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:36.967482 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:36.967276    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:38.212203 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.211727    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cbjgx" event={"ID":"2337c617-d3c5-417f-b2c8-4c8328a04a58","Type":"ContainerStarted","Data":"42afd743f0618b7e2b80df616f1e4201226830ecf3a063b5d06f88f7bfe20125"}
Sep 12 00:43:38.213573 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.213543    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g5brn" event={"ID":"cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c","Type":"ContainerStarted","Data":"a50115d7e85c85df9f1761a8bc4f6a4b9701c7d3e1e151ff21579089e0374cc6"}
Sep 12 00:43:38.217458 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.217437    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g" event={"ID":"937692a3-80e6-477f-a7f9-7cdbaa4ba628","Type":"ContainerStarted","Data":"7c4e07ccf6a014be61bb847887e1b42e198a77ffcff7e063275bb2f520390726"}
Sep 12 00:43:38.217557 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.217501    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g" event={"ID":"937692a3-80e6-477f-a7f9-7cdbaa4ba628","Type":"ContainerStarted","Data":"f25aff9e727de737c2b9f4437a36510593561037b72de393d3f392d090419ded"}
Sep 12 00:43:38.217557 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.217538    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g" event={"ID":"937692a3-80e6-477f-a7f9-7cdbaa4ba628","Type":"ContainerStarted","Data":"dfc1375491e17bdf4e1706abf3560ad4758c3ed1a4b28f4a4fb5032099bdeb4c"}
Sep 12 00:43:38.217672 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.217580    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g" event={"ID":"937692a3-80e6-477f-a7f9-7cdbaa4ba628","Type":"ContainerStarted","Data":"3a3018d34d1e3f00cf6c938c3ade90f1cd7dbf7a4a04f12b9e1de3aded1b8885"}
Sep 12 00:43:38.217672 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.217614    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g" event={"ID":"937692a3-80e6-477f-a7f9-7cdbaa4ba628","Type":"ContainerStarted","Data":"4fe70d3bbf67a254a94622a9941b6d3befe50011d6a68e3462e2c21139217036"}
Sep 12 00:43:38.217672 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.217667    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g" event={"ID":"937692a3-80e6-477f-a7f9-7cdbaa4ba628","Type":"ContainerStarted","Data":"bf6da174b7c59a46bda0a095590add231082249c3b011dabe514d475d90bc51e"}
Sep 12 00:43:38.219200 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.219160    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-121.ec2.internal" event={"ID":"ad9778864c1c592078495eb7196d09e3","Type":"ContainerStarted","Data":"4923eb3617996336ceb9a7169234b27fcd97e5091cce347d9e8ef7fbc9de3b4e"}
Sep 12 00:43:38.220879 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.220825    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm" event={"ID":"5128643f-5556-48f8-95d9-d82dfa1c3cae","Type":"ContainerStarted","Data":"892844d20d963f9577796cabe3ece733d1a9b282390022495ae81240dab26650"}
Sep 12 00:43:38.222456 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.222430    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rnzw4" event={"ID":"d0f1ca3a-739a-4deb-8f9e-de39d894d430","Type":"ContainerStarted","Data":"9dddfe18e08a71d398b21df7eb2975dd7065a992544334ce8193d6f1d1bc4a23"}
Sep 12 00:43:38.223978 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.223936    2567 generic.go:356] "Generic (PLEG): container finished" podID="f2ef7878-3d24-42f5-9708-2fe05bbe4bdf" containerID="2e088b859646dbd92328ae2bb9daa3b64d07160c37664323ba395ad6b993b74b" exitCode=0
Sep 12 00:43:38.224088 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.224029    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg489" event={"ID":"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf","Type":"ContainerDied","Data":"2e088b859646dbd92328ae2bb9daa3b64d07160c37664323ba395ad6b993b74b"}
Sep 12 00:43:38.225862 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.225835    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mvtjp" event={"ID":"5bf0d680-e19c-4892-8652-58f49ed9d3b3","Type":"ContainerStarted","Data":"d9c998995c815b097533f0cdaa936e297d3b95aa7bfeeaf59b7f39c4081f4176"}
Sep 12 00:43:38.227813 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.227766    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cbjgx" podStartSLOduration=1.445372527 podStartE2EDuration="21.227748689s" totalImagesPullingTime="19.782376162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:38.227156699 +0000 UTC m=+22.787608358" watchObservedRunningTime="2026-09-12 00:43:38.227748689 +0000 UTC m=+22.788200361"
Sep 12 00:43:38.228405 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.228320    2567 generic.go:356] "Generic (PLEG): container finished" podID="9fd7b5f142d9f49960e1ce88e20ea881" containerID="144d55fe9fea96749ea1a13851486ff77ff57626f5a5dafed35f2b4a9e0dfb87" exitCode=0
Sep 12 00:43:38.228405 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.228357    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal" event={"ID":"9fd7b5f142d9f49960e1ce88e20ea881","Type":"ContainerDied","Data":"144d55fe9fea96749ea1a13851486ff77ff57626f5a5dafed35f2b4a9e0dfb87"}
Sep 12 00:43:38.228554 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.228507    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal"
Sep 12 00:43:38.230261 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.230242    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h2vvz" event={"ID":"1bb2ff8c-97ad-4543-975f-caed929a31c8","Type":"ContainerStarted","Data":"24a66e0453035a15cf2abdebbbaf1a752164a0ba5f6f204024872985ed86cdf5"}
Sep 12 00:43:38.244547 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.244472    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-121.ec2.internal" podStartSLOduration=21.244458996 podStartE2EDuration="21.244458996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:38.243875478 +0000 UTC m=+22.804327148" watchObservedRunningTime="2026-09-12 00:43:38.244458996 +0000 UTC m=+22.804910664"
Sep 12 00:43:38.246825 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.246794    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 12 00:43:38.247538 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.247517    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal"]
Sep 12 00:43:38.259686 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.259650    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g5brn" podStartSLOduration=2.464578573 podStartE2EDuration="22.259635635s" totalImagesPullingTime="19.795057062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:38.259197539 +0000 UTC m=+22.819649208" watchObservedRunningTime="2026-09-12 00:43:38.259635635 +0000 UTC m=+22.820087303"
Sep 12 00:43:38.275162 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.275127    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mvtjp" podStartSLOduration=2.4489832910000002 podStartE2EDuration="22.275109282s" totalImagesPullingTime="19.826125991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:38.274840813 +0000 UTC m=+22.835292503" watchObservedRunningTime="2026-09-12 00:43:38.275109282 +0000 UTC m=+22.835560949"
Sep 12 00:43:38.333768 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.333728    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rnzw4" podStartSLOduration=2.260013346 podStartE2EDuration="22.333713015s" totalImagesPullingTime="20.073699669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:38.319430553 +0000 UTC m=+22.879882223" watchObservedRunningTime="2026-09-12 00:43:38.333713015 +0000 UTC m=+22.894164683"
Sep 12 00:43:38.354854 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.354814    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h2vvz" podStartSLOduration=1.593880985 podStartE2EDuration="21.354800167s" totalImagesPullingTime="19.760919182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:38.354515093 +0000 UTC m=+22.914966760" watchObservedRunningTime="2026-09-12 00:43:38.354800167 +0000 UTC m=+22.915251835"
Sep 12 00:43:38.393982 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.393924    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:38.456523 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.456490    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:38.518225 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.518158    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:38.915896 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.915831    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g5brn"
Sep 12 00:43:38.967079 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.967049    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:38.967079 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.967074    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:38.967256 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:38.967160    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:38.967256 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:38.967166    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:38.967256 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:38.967222    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:38.967368 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:38.967290    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:39.177580 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:39.177523    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 12 00:43:39.234524 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:39.234486    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm" event={"ID":"5128643f-5556-48f8-95d9-d82dfa1c3cae","Type":"ContainerStarted","Data":"ab56651c51969247fdf368849ecfb2d20836bbe82a61cf8bce5499aaf5050e96"}
Sep 12 00:43:39.236507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:39.236475    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal" event={"ID":"9fd7b5f142d9f49960e1ce88e20ea881","Type":"ContainerStarted","Data":"80ac23df4de9d6ff6b41c93154a5be7230b73f778728bfb903c0bddd9d1c25b0"}
Sep 12 00:43:39.238270 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:39.238238    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5k6jv" event={"ID":"0f47656b-bd7c-4aa2-9943-3de60c25f96b","Type":"ContainerStarted","Data":"e7fc1755c91630d5a8788ddb7fcca87ee80c8277adc26140702c225bd1889e68"}
Sep 12 00:43:39.253113 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:39.253059    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-121.ec2.internal" podStartSLOduration=1.253045157 podStartE2EDuration="1.253045157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:39.252480845 +0000 UTC m=+23.812932513" watchObservedRunningTime="2026-09-12 00:43:39.253045157 +0000 UTC m=+23.813496825"
Sep 12 00:43:39.267874 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:39.267834    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5k6jv" podStartSLOduration=3.454279102 podStartE2EDuration="23.267818295s" totalImagesPullingTime="19.813539193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:39.267098153 +0000 UTC m=+23.827549823" watchObservedRunningTime="2026-09-12 00:43:39.267818295 +0000 UTC m=+23.828269967"
Sep 12 00:43:39.386968 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:39.386926    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:39.446297 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:39.446226    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:39.909250 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:39.908906    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-12T00:43:39.177543617Z","UUID":"17cd3a7a-dde6-41c6-aa04-c64382e54c93","Handler":null,"Name":"","Endpoint":""}
Sep 12 00:43:39.910634 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:39.910613    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 12 00:43:39.910634 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:39.910646    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 12 00:43:40.244684 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:40.244604    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g" event={"ID":"937692a3-80e6-477f-a7f9-7cdbaa4ba628","Type":"ContainerStarted","Data":"bb574b8ba244605be910253fb64b40bb02613b7c2a82a2f87676c78d160abec6"}
Sep 12 00:43:40.967346 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:40.967263    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:40.967495 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:40.967367    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:40.967495 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:40.967373    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:40.967495 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:40.967410    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:40.967495 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:40.967479    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:40.967656 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:40.967532    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:41.249305 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:41.249223    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm" event={"ID":"5128643f-5556-48f8-95d9-d82dfa1c3cae","Type":"ContainerStarted","Data":"f196d38ec33b11294209bc091a8a54227f426b9e70c639522c4f9bb3c2d5b370"}
Sep 12 00:43:41.267812 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:41.267761    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l4zm" podStartSLOduration=1.796745761 podStartE2EDuration="24.2677431s" totalImagesPullingTime="22.470997339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:43:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:41.26773285 +0000 UTC m=+25.828184543" watchObservedRunningTime="2026-09-12 00:43:41.2677431 +0000 UTC m=+25.828194765"
Sep 12 00:43:42.216960 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:42.216914    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g5brn"
Sep 12 00:43:42.217595 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:42.217570    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g5brn"
Sep 12 00:43:42.252539 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:42.252513    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g5brn"
Sep 12 00:43:42.322049 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:42.321979    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:42.967670 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:42.967643    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:42.967826 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:42.967644    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:42.967826 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:42.967777    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:42.967826 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:42.967644    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:42.967926 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:42.967814    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:42.967926 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:42.967893    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:43.258318 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:43.258245    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g" event={"ID":"937692a3-80e6-477f-a7f9-7cdbaa4ba628","Type":"ContainerStarted","Data":"19df345d9adbb7586ecf04083c5ad167dd30ed939f828ab33952795197343aba"}
Sep 12 00:43:43.258682 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:43.258536    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:43.258682 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:43.258599    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:43.258682 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:43.258642    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:43.276292 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:43.276129    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:43.277015 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:43.276993    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:43:43.288167 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:43.287341    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g" podStartSLOduration=7.445039605 podStartE2EDuration="27.287322885s" totalImagesPullingTime="19.84228328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:43.287004026 +0000 UTC m=+27.847455682" watchObservedRunningTime="2026-09-12 00:43:43.287322885 +0000 UTC m=+27.847774556"
Sep 12 00:43:44.261607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:44.261576    2567 generic.go:356] "Generic (PLEG): container finished" podID="f2ef7878-3d24-42f5-9708-2fe05bbe4bdf" containerID="4727cd737593a1b137c5a1afdf917332ede6e4748085e5f87954e588f4dea5c9" exitCode=0
Sep 12 00:43:44.262146 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:44.261676    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg489" event={"ID":"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf","Type":"ContainerDied","Data":"4727cd737593a1b137c5a1afdf917332ede6e4748085e5f87954e588f4dea5c9"}
Sep 12 00:43:44.967799 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:44.967766    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:44.968009 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:44.967766    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:44.968009 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:44.967896    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:44.968009 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:44.967999    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:44.968164 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:44.967766    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:44.968164 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:44.968119    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:45.007111 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:45.007076    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5k8lk"]
Sep 12 00:43:45.010821 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:45.010771    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bdgdv"]
Sep 12 00:43:45.011271 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:45.011245    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sdhm4"]
Sep 12 00:43:45.265489 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:45.265302    2567 generic.go:356] "Generic (PLEG): container finished" podID="f2ef7878-3d24-42f5-9708-2fe05bbe4bdf" containerID="95a28673bf11baacd3978aae02da81f3122df8b12cabe19b568ca8562cff1160" exitCode=0
Sep 12 00:43:45.266323 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:45.265563    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:45.266323 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:45.265376    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg489" event={"ID":"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf","Type":"ContainerDied","Data":"95a28673bf11baacd3978aae02da81f3122df8b12cabe19b568ca8562cff1160"}
Sep 12 00:43:45.266323 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:45.265668    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:45.266323 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:45.265677    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:45.266323 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:45.266088    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:45.266323 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:45.266104    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:45.266323 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:45.266185    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:46.270887 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:46.270857    2567 generic.go:356] "Generic (PLEG): container finished" podID="f2ef7878-3d24-42f5-9708-2fe05bbe4bdf" containerID="07441ddbfa3a903cee1e099d130ff41bff671879cf3c63303940429ed24d72d6" exitCode=0
Sep 12 00:43:46.271276 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:46.270903    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg489" event={"ID":"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf","Type":"ContainerDied","Data":"07441ddbfa3a903cee1e099d130ff41bff671879cf3c63303940429ed24d72d6"}
Sep 12 00:43:46.967736 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:46.967663    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:46.967890 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:46.967663    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:46.967890 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:46.967784    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:46.967890 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:46.967664    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:46.967890 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:46.967846    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:46.968113 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:46.967922    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:48.585794 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:48.585704    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs\") pod \"network-metrics-daemon-sdhm4\" (UID: \"079a162f-f33c-4911-8385-e6d87ccd2231\") " pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:48.586296 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:48.585849    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:48.586296 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:48.585927    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs podName:079a162f-f33c-4911-8385-e6d87ccd2231 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:20.585905017 +0000 UTC m=+65.146356664 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs") pod "network-metrics-daemon-sdhm4" (UID: "079a162f-f33c-4911-8385-e6d87ccd2231") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:48.686163 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:48.686124    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:48.686340 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:48.686274    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:48.686407 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:48.686366    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret podName:d5d08bf0-f998-4798-b16c-ec2c5cecbe21 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:20.686346498 +0000 UTC m=+65.246798143 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret") pod "global-pull-secret-syncer-5k8lk" (UID: "d5d08bf0-f998-4798-b16c-ec2c5cecbe21") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:48.786814 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:48.786780    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlkbs\" (UniqueName: \"kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs\") pod \"network-check-target-bdgdv\" (UID: \"32f044d9-76e0-48fd-bd47-9f829d5a74d5\") " pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:48.786995 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:48.786917    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:43:48.786995 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:48.786931    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:43:48.786995 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:48.786939    2567 projected.go:196] Error preparing data for projected volume kube-api-access-hlkbs for pod openshift-network-diagnostics/network-check-target-bdgdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:48.787100 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:48.787008    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs podName:32f044d9-76e0-48fd-bd47-9f829d5a74d5 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:20.786994937 +0000 UTC m=+65.347446582 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hlkbs" (UniqueName: "kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs") pod "network-check-target-bdgdv" (UID: "32f044d9-76e0-48fd-bd47-9f829d5a74d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:48.967634 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:48.967548    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:48.967634 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:48.967587    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:48.967848 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:48.967679    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:48.967848 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:48.967693    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdhm4" podUID="079a162f-f33c-4911-8385-e6d87ccd2231"
Sep 12 00:43:48.967848 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:48.967773    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5k8lk" podUID="d5d08bf0-f998-4798-b16c-ec2c5cecbe21"
Sep 12 00:43:48.968012 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:48.967942    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdgdv" podUID="32f044d9-76e0-48fd-bd47-9f829d5a74d5"
Sep 12 00:43:50.234436 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.234401    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-121.ec2.internal" event="NodeReady"
Sep 12 00:43:50.234913 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.234580    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 12 00:43:50.305533 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.305500    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dsbc2"]
Sep 12 00:43:50.356247 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.356178    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dsbc2"]
Sep 12 00:43:50.356247 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.356217    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6z6bf"]
Sep 12 00:43:50.356425 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.356321    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:50.358773 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.358749    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 12 00:43:50.358916 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.358774    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 12 00:43:50.358916 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.358852    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2gxdg\""
Sep 12 00:43:50.371390 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.371369    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6z6bf"]
Sep 12 00:43:50.371586 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.371471    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6z6bf"
Sep 12 00:43:50.374110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.374093    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 12 00:43:50.374528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.374499    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 12 00:43:50.374607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.374540    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qtwt4\""
Sep 12 00:43:50.374607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.374554    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 12 00:43:50.471648 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.471586    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:50.500631 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.500594    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-config-volume\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:50.500813 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.500636    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ee67b56-1998-4545-9ceb-8597c6101dab-cert\") pod \"ingress-canary-6z6bf\" (UID: \"2ee67b56-1998-4545-9ceb-8597c6101dab\") " pod="openshift-ingress-canary/ingress-canary-6z6bf"
Sep 12 00:43:50.500813 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.500669    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcdth\" (UniqueName: \"kubernetes.io/projected/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-kube-api-access-xcdth\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:50.500813 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.500723    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wpr\" (UniqueName: \"kubernetes.io/projected/2ee67b56-1998-4545-9ceb-8597c6101dab-kube-api-access-x5wpr\") pod \"ingress-canary-6z6bf\" (UID: \"2ee67b56-1998-4545-9ceb-8597c6101dab\") " pod="openshift-ingress-canary/ingress-canary-6z6bf"
Sep 12 00:43:50.500813 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.500784    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:50.500813 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.500800    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-tmp-dir\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:50.571864 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.571824    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:50.602116 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.602090    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:50.602116 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.602119    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-tmp-dir\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:50.602366 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.602164    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-config-volume\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:50.602366 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.602185    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ee67b56-1998-4545-9ceb-8597c6101dab-cert\") pod \"ingress-canary-6z6bf\" (UID: \"2ee67b56-1998-4545-9ceb-8597c6101dab\") " pod="openshift-ingress-canary/ingress-canary-6z6bf"
Sep 12 00:43:50.602366 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.602205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcdth\" (UniqueName: \"kubernetes.io/projected/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-kube-api-access-xcdth\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:50.602366 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.602233    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wpr\" (UniqueName: \"kubernetes.io/projected/2ee67b56-1998-4545-9ceb-8597c6101dab-kube-api-access-x5wpr\") pod \"ingress-canary-6z6bf\" (UID: \"2ee67b56-1998-4545-9ceb-8597c6101dab\") " pod="openshift-ingress-canary/ingress-canary-6z6bf"
Sep 12 00:43:50.602366 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:50.602266    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:43:50.602366 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:50.602346    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls podName:d07d13ab-1c44-499a-aa9f-ebed6bbe188e nodeName:}" failed. No retries permitted until 2026-09-12 00:43:51.102325096 +0000 UTC m=+35.662776765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls") pod "dns-default-dsbc2" (UID: "d07d13ab-1c44-499a-aa9f-ebed6bbe188e") : secret "dns-default-metrics-tls" not found
Sep 12 00:43:50.602663 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.602452    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-tmp-dir\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:50.602847 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.602827    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-config-volume\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:50.606344 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.606304    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ee67b56-1998-4545-9ceb-8597c6101dab-cert\") pod \"ingress-canary-6z6bf\" (UID: \"2ee67b56-1998-4545-9ceb-8597c6101dab\") " pod="openshift-ingress-canary/ingress-canary-6z6bf"
Sep 12 00:43:50.613988 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.613925    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wpr\" (UniqueName: \"kubernetes.io/projected/2ee67b56-1998-4545-9ceb-8597c6101dab-kube-api-access-x5wpr\") pod \"ingress-canary-6z6bf\" (UID: \"2ee67b56-1998-4545-9ceb-8597c6101dab\") " pod="openshift-ingress-canary/ingress-canary-6z6bf"
Sep 12 00:43:50.614571 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.614549    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcdth\" (UniqueName: \"kubernetes.io/projected/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-kube-api-access-xcdth\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:50.681662 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.681625    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6z6bf"
Sep 12 00:43:50.967929 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.967903    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:43:50.968072 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.967903    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:43:50.968133 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.968112    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:43:50.971514 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.971107    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 12 00:43:50.971514 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.971467    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 12 00:43:50.971879 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.971795    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-42kqk\""
Sep 12 00:43:50.971879 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.971803    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 12 00:43:50.972077 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.971932    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 12 00:43:50.972077 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.971942    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pfcnt\""
Sep 12 00:43:50.991382 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:50.991347    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6z6bf"]
Sep 12 00:43:50.997081 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:43:50.997059    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee67b56_1998_4545_9ceb_8597c6101dab.slice/crio-c5836882b074e6da0f5d86a3125733d272f6b57d57b45a06a73b6776bb43abaa WatchSource:0}: Error finding container c5836882b074e6da0f5d86a3125733d272f6b57d57b45a06a73b6776bb43abaa: Status 404 returned error can't find the container with id c5836882b074e6da0f5d86a3125733d272f6b57d57b45a06a73b6776bb43abaa
Sep 12 00:43:51.106918 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:51.106890    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:51.107081 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:51.107063    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:43:51.107146 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:51.107132    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls podName:d07d13ab-1c44-499a-aa9f-ebed6bbe188e nodeName:}" failed. No retries permitted until 2026-09-12 00:43:52.107113378 +0000 UTC m=+36.667565026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls") pod "dns-default-dsbc2" (UID: "d07d13ab-1c44-499a-aa9f-ebed6bbe188e") : secret "dns-default-metrics-tls" not found
Sep 12 00:43:51.284922 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:51.284888    2567 generic.go:356] "Generic (PLEG): container finished" podID="f2ef7878-3d24-42f5-9708-2fe05bbe4bdf" containerID="75f12ff4ab6b23a61710316b93878d71177c9a6bbb36d0cf870e9f69d8ac2665" exitCode=0
Sep 12 00:43:51.285421 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:51.284979    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg489" event={"ID":"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf","Type":"ContainerDied","Data":"75f12ff4ab6b23a61710316b93878d71177c9a6bbb36d0cf870e9f69d8ac2665"}
Sep 12 00:43:51.286207 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:51.286187    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6z6bf" event={"ID":"2ee67b56-1998-4545-9ceb-8597c6101dab","Type":"ContainerStarted","Data":"c5836882b074e6da0f5d86a3125733d272f6b57d57b45a06a73b6776bb43abaa"}
Sep 12 00:43:52.115224 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:52.115185    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:52.115410 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:52.115347    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:43:52.115474 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:52.115423    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls podName:d07d13ab-1c44-499a-aa9f-ebed6bbe188e nodeName:}" failed. No retries permitted until 2026-09-12 00:43:54.115402395 +0000 UTC m=+38.675854060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls") pod "dns-default-dsbc2" (UID: "d07d13ab-1c44-499a-aa9f-ebed6bbe188e") : secret "dns-default-metrics-tls" not found
Sep 12 00:43:52.291811 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:52.291775    2567 generic.go:356] "Generic (PLEG): container finished" podID="f2ef7878-3d24-42f5-9708-2fe05bbe4bdf" containerID="a0fe4f2efd346b852d02061347a1e11d4c283111523ae64b47de04f8c746cfe9" exitCode=0
Sep 12 00:43:52.292384 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:52.291823    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg489" event={"ID":"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf","Type":"ContainerDied","Data":"a0fe4f2efd346b852d02061347a1e11d4c283111523ae64b47de04f8c746cfe9"}
Sep 12 00:43:53.297812 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:53.297784    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg489" event={"ID":"f2ef7878-3d24-42f5-9708-2fe05bbe4bdf","Type":"ContainerStarted","Data":"1209c2af1add3ab04a07a65c40cb901b611fb4b2ae4055beabb15d0ff16eec3f"}
Sep 12 00:43:53.322635 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:53.322588    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qg489" podStartSLOduration=6.322913235 podStartE2EDuration="37.322573375s" totalImagesPullingTime="30.99966014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:43:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:53.321367048 +0000 UTC m=+37.881818741" watchObservedRunningTime="2026-09-12 00:43:53.322573375 +0000 UTC m=+37.883025036"
Sep 12 00:43:53.454680 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:53.454459    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:53.557776 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:53.557711    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:54.132318 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:54.132278    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:54.132450 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:54.132428    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:43:54.132503 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:54.132492    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls podName:d07d13ab-1c44-499a-aa9f-ebed6bbe188e nodeName:}" failed. No retries permitted until 2026-09-12 00:43:58.13247686 +0000 UTC m=+42.692928507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls") pod "dns-default-dsbc2" (UID: "d07d13ab-1c44-499a-aa9f-ebed6bbe188e") : secret "dns-default-metrics-tls" not found
Sep 12 00:43:54.302028 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:54.301989    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6z6bf" event={"ID":"2ee67b56-1998-4545-9ceb-8597c6101dab","Type":"ContainerStarted","Data":"bc03663cf2a6de59652eccdfcc2e81c8433f9f6a4fc69aebcd61eb3b2fb3f404"}
Sep 12 00:43:54.321533 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:54.321490    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6z6bf" podStartSLOduration=1.865327645 podStartE2EDuration="4.321477008s" totalImagesPullingTime="2.456149363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:54.320725459 +0000 UTC m=+38.881177129" watchObservedRunningTime="2026-09-12 00:43:54.321477008 +0000 UTC m=+38.881928675"
Sep 12 00:43:54.912056 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:54.911980    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:55.027489 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:55.027456    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:55.094827 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:55.094795    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:58.008456 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:58.008422    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:58.161271 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:58.161239    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:43:58.161404 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:58.161385    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:43:58.161455 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:43:58.161444    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls podName:d07d13ab-1c44-499a-aa9f-ebed6bbe188e nodeName:}" failed. No retries permitted until 2026-09-12 00:44:06.161430987 +0000 UTC m=+50.721882632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls") pod "dns-default-dsbc2" (UID: "d07d13ab-1c44-499a-aa9f-ebed6bbe188e") : secret "dns-default-metrics-tls" not found
Sep 12 00:43:59.127169 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:43:59.127129    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:06.219351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:06.219312    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:44:06.219792 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:06.219431    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:44:06.219792 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:06.219493    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls podName:d07d13ab-1c44-499a-aa9f-ebed6bbe188e nodeName:}" failed. No retries permitted until 2026-09-12 00:44:22.219480875 +0000 UTC m=+66.779932522 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls") pod "dns-default-dsbc2" (UID: "d07d13ab-1c44-499a-aa9f-ebed6bbe188e") : secret "dns-default-metrics-tls" not found
Sep 12 00:44:15.276302 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:15.276273    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srp9g"
Sep 12 00:44:15.400213 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:15.400180    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:15.462914 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:15.462887    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:20.609399 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:20.609354    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs\") pod \"network-metrics-daemon-sdhm4\" (UID: \"079a162f-f33c-4911-8385-e6d87ccd2231\") " pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:44:20.612351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:20.612329    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 12 00:44:20.620573 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:20.620550    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 12 00:44:20.620666 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:20.620636    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs podName:079a162f-f33c-4911-8385-e6d87ccd2231 nodeName:}" failed. No retries permitted until 2026-09-12 00:45:24.62061465 +0000 UTC m=+129.181066302 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs") pod "network-metrics-daemon-sdhm4" (UID: "079a162f-f33c-4911-8385-e6d87ccd2231") : secret "metrics-daemon-secret" not found
Sep 12 00:44:20.710162 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:20.710129    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:44:20.713085 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:20.713069    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 12 00:44:20.724139 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:20.724112    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d08bf0-f998-4798-b16c-ec2c5cecbe21-original-pull-secret\") pod \"global-pull-secret-syncer-5k8lk\" (UID: \"d5d08bf0-f998-4798-b16c-ec2c5cecbe21\") " pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:44:20.810673 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:20.810638    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlkbs\" (UniqueName: \"kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs\") pod \"network-check-target-bdgdv\" (UID: \"32f044d9-76e0-48fd-bd47-9f829d5a74d5\") " pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:44:20.813546 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:20.813526    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 12 00:44:20.823618 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:20.823597    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 12 00:44:20.834857 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:20.834829    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlkbs\" (UniqueName: \"kubernetes.io/projected/32f044d9-76e0-48fd-bd47-9f829d5a74d5-kube-api-access-hlkbs\") pod \"network-check-target-bdgdv\" (UID: \"32f044d9-76e0-48fd-bd47-9f829d5a74d5\") " pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:44:20.984452 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:20.984418    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pfcnt\""
Sep 12 00:44:20.992588 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:20.992565    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:44:20.998875 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:20.998856    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5k8lk"
Sep 12 00:44:21.296797 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:21.296727    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bdgdv"]
Sep 12 00:44:21.299803 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:44:21.299777    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f044d9_76e0_48fd_bd47_9f829d5a74d5.slice/crio-0fe3b3c01c96920fb86604b730d5f614882525c87b9d30b83bcfc4402eb2d297 WatchSource:0}: Error finding container 0fe3b3c01c96920fb86604b730d5f614882525c87b9d30b83bcfc4402eb2d297: Status 404 returned error can't find the container with id 0fe3b3c01c96920fb86604b730d5f614882525c87b9d30b83bcfc4402eb2d297
Sep 12 00:44:21.319567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:21.319538    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5k8lk"]
Sep 12 00:44:21.322995 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:44:21.322972    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d08bf0_f998_4798_b16c_ec2c5cecbe21.slice/crio-11abd6f789cbc5cfcbb2f86f848961497e5d420b648be94a7d734964410695d9 WatchSource:0}: Error finding container 11abd6f789cbc5cfcbb2f86f848961497e5d420b648be94a7d734964410695d9: Status 404 returned error can't find the container with id 11abd6f789cbc5cfcbb2f86f848961497e5d420b648be94a7d734964410695d9
Sep 12 00:44:21.364557 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:21.364517    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5k8lk" event={"ID":"d5d08bf0-f998-4798-b16c-ec2c5cecbe21","Type":"ContainerStarted","Data":"11abd6f789cbc5cfcbb2f86f848961497e5d420b648be94a7d734964410695d9"}
Sep 12 00:44:21.365573 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:21.365547    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bdgdv" event={"ID":"32f044d9-76e0-48fd-bd47-9f829d5a74d5","Type":"ContainerStarted","Data":"0fe3b3c01c96920fb86604b730d5f614882525c87b9d30b83bcfc4402eb2d297"}
Sep 12 00:44:22.220537 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:22.220498    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:44:22.221026 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:22.220656    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:44:22.221026 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:22.220725    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls podName:d07d13ab-1c44-499a-aa9f-ebed6bbe188e nodeName:}" failed. No retries permitted until 2026-09-12 00:44:54.220710414 +0000 UTC m=+98.781162060 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls") pod "dns-default-dsbc2" (UID: "d07d13ab-1c44-499a-aa9f-ebed6bbe188e") : secret "dns-default-metrics-tls" not found
Sep 12 00:44:26.379672 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:26.379575    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5k8lk" event={"ID":"d5d08bf0-f998-4798-b16c-ec2c5cecbe21","Type":"ContainerStarted","Data":"6f77990688e0b7999058763f738adb8bef3323cd2a71e4e9769846f9b5bb6f05"}
Sep 12 00:44:26.381039 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:26.381014    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bdgdv" event={"ID":"32f044d9-76e0-48fd-bd47-9f829d5a74d5","Type":"ContainerStarted","Data":"fe0938477fc4dfd1b6aaef53d71499771cc111b0a501b479ed948262b0822e4f"}
Sep 12 00:44:26.381174 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:26.381158    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:44:26.396397 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:26.396347    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5k8lk" podStartSLOduration=64.739677887 podStartE2EDuration="1m9.396333088s" totalImagesPullingTime="4.656655201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:26.395653724 +0000 UTC m=+70.956105393" watchObservedRunningTime="2026-09-12 00:44:26.396333088 +0000 UTC m=+70.956784755"
Sep 12 00:44:26.413124 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:26.413078    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bdgdv" podStartSLOduration=65.7387318 podStartE2EDuration="1m10.413065955s" totalImagesPullingTime="4.674334155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:26.411908822 +0000 UTC m=+70.972360500" watchObservedRunningTime="2026-09-12 00:44:26.413065955 +0000 UTC m=+70.973517624"
Sep 12 00:44:37.786220 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.786103    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz"]
Sep 12 00:44:37.789120 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.789100    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz"
Sep 12 00:44:37.792656 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.792636    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 12 00:44:37.792746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.792673    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vbwlw\""
Sep 12 00:44:37.792746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.792679    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 12 00:44:37.793298 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.792687    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:44:37.793298 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.793161    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 12 00:44:37.799938 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.799916    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz"]
Sep 12 00:44:37.817582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.817534    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlfm2\" (UniqueName: \"kubernetes.io/projected/48b0b2b4-0df2-410f-89d0-705dd447dc8a-kube-api-access-wlfm2\") pod \"service-ca-operator-cf7c776b7-g22zz\" (UID: \"48b0b2b4-0df2-410f-89d0-705dd447dc8a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz"
Sep 12 00:44:37.817731 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.817630    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/48b0b2b4-0df2-410f-89d0-705dd447dc8a-config\") pod \"service-ca-operator-cf7c776b7-g22zz\" (UID: \"48b0b2b4-0df2-410f-89d0-705dd447dc8a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz"
Sep 12 00:44:37.817731 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.817674    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/48b0b2b4-0df2-410f-89d0-705dd447dc8a-serving-cert\") pod \"service-ca-operator-cf7c776b7-g22zz\" (UID: \"48b0b2b4-0df2-410f-89d0-705dd447dc8a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz"
Sep 12 00:44:37.893229 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.893192    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-v9xsz"]
Sep 12 00:44:37.896036 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.895985    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-v9xsz"
Sep 12 00:44:37.898712 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.898688    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 12 00:44:37.899067 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.899051    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-f65pq\""
Sep 12 00:44:37.899123 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.899051    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:44:37.901905 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.901886    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt"]
Sep 12 00:44:37.904686 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.904670    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"]
Sep 12 00:44:37.904814 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.904799    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt"
Sep 12 00:44:37.907485 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.907467    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:44:37.907603 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.907585    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 12 00:44:37.907669 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.907588    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 12 00:44:37.907669 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.907627    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"
Sep 12 00:44:37.907765 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.907742    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pwnqd\""
Sep 12 00:44:37.909796 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.909777    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:44:37.910244 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.910228    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 12 00:44:37.911076 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.911057    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 12 00:44:37.911172 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.911108    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-sq2bj\""
Sep 12 00:44:37.918641 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.918622    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlfm2\" (UniqueName: \"kubernetes.io/projected/48b0b2b4-0df2-410f-89d0-705dd447dc8a-kube-api-access-wlfm2\") pod \"service-ca-operator-cf7c776b7-g22zz\" (UID: \"48b0b2b4-0df2-410f-89d0-705dd447dc8a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz"
Sep 12 00:44:37.918719 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.918652    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6rx9\" (UniqueName: \"kubernetes.io/projected/ae656cec-be27-43b5-8123-f2bb132e6de3-kube-api-access-x6rx9\") pod \"cluster-samples-operator-86499c89c4-kkwg8\" (UID: \"ae656cec-be27-43b5-8123-f2bb132e6de3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"
Sep 12 00:44:37.918719 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.918674    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kkwg8\" (UID: \"ae656cec-be27-43b5-8123-f2bb132e6de3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"
Sep 12 00:44:37.918719 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.918696    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8stl5\" (UniqueName: \"kubernetes.io/projected/a58e1e17-be8b-47d5-bdee-cd38e5554c44-kube-api-access-8stl5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5pnzt\" (UID: \"a58e1e17-be8b-47d5-bdee-cd38e5554c44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt"
Sep 12 00:44:37.918876 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.918721    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a58e1e17-be8b-47d5-bdee-cd38e5554c44-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5pnzt\" (UID: \"a58e1e17-be8b-47d5-bdee-cd38e5554c44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt"
Sep 12 00:44:37.918876 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.918739    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a58e1e17-be8b-47d5-bdee-cd38e5554c44-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5pnzt\" (UID: \"a58e1e17-be8b-47d5-bdee-cd38e5554c44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt"
Sep 12 00:44:37.918876 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.918812    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/48b0b2b4-0df2-410f-89d0-705dd447dc8a-config\") pod \"service-ca-operator-cf7c776b7-g22zz\" (UID: \"48b0b2b4-0df2-410f-89d0-705dd447dc8a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz"
Sep 12 00:44:37.918876 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.918845    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/48b0b2b4-0df2-410f-89d0-705dd447dc8a-serving-cert\") pod \"service-ca-operator-cf7c776b7-g22zz\" (UID: \"48b0b2b4-0df2-410f-89d0-705dd447dc8a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz"
Sep 12 00:44:37.919049 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.918879    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg2j7\" (UniqueName: \"kubernetes.io/projected/08374dad-0ebd-4626-8470-566aa010ba48-kube-api-access-qg2j7\") pod \"volume-data-source-validator-86695b7cc7-v9xsz\" (UID: \"08374dad-0ebd-4626-8470-566aa010ba48\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-v9xsz"
Sep 12 00:44:37.919264 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.919247    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/48b0b2b4-0df2-410f-89d0-705dd447dc8a-config\") pod \"service-ca-operator-cf7c776b7-g22zz\" (UID: \"48b0b2b4-0df2-410f-89d0-705dd447dc8a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz"
Sep 12 00:44:37.920107 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.920087    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 12 00:44:37.921101 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.921080    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/48b0b2b4-0df2-410f-89d0-705dd447dc8a-serving-cert\") pod \"service-ca-operator-cf7c776b7-g22zz\" (UID: \"48b0b2b4-0df2-410f-89d0-705dd447dc8a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz"
Sep 12 00:44:37.932052 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.932029    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-v9xsz"]
Sep 12 00:44:37.932837 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.932814    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"]
Sep 12 00:44:37.933399 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.933382    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt"]
Sep 12 00:44:37.952986 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:37.952937    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlfm2\" (UniqueName: \"kubernetes.io/projected/48b0b2b4-0df2-410f-89d0-705dd447dc8a-kube-api-access-wlfm2\") pod \"service-ca-operator-cf7c776b7-g22zz\" (UID: \"48b0b2b4-0df2-410f-89d0-705dd447dc8a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz"
Sep 12 00:44:38.019991 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.019941    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6rx9\" (UniqueName: \"kubernetes.io/projected/ae656cec-be27-43b5-8123-f2bb132e6de3-kube-api-access-x6rx9\") pod \"cluster-samples-operator-86499c89c4-kkwg8\" (UID: \"ae656cec-be27-43b5-8123-f2bb132e6de3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"
Sep 12 00:44:38.020152 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.019999    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kkwg8\" (UID: \"ae656cec-be27-43b5-8123-f2bb132e6de3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"
Sep 12 00:44:38.020152 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.020036    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8stl5\" (UniqueName: \"kubernetes.io/projected/a58e1e17-be8b-47d5-bdee-cd38e5554c44-kube-api-access-8stl5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5pnzt\" (UID: \"a58e1e17-be8b-47d5-bdee-cd38e5554c44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt"
Sep 12 00:44:38.020152 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.020065    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a58e1e17-be8b-47d5-bdee-cd38e5554c44-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5pnzt\" (UID: \"a58e1e17-be8b-47d5-bdee-cd38e5554c44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt"
Sep 12 00:44:38.020152 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.020089    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a58e1e17-be8b-47d5-bdee-cd38e5554c44-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5pnzt\" (UID: \"a58e1e17-be8b-47d5-bdee-cd38e5554c44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt"
Sep 12 00:44:38.020152 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:38.020099    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 12 00:44:38.020152 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.020138    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg2j7\" (UniqueName: \"kubernetes.io/projected/08374dad-0ebd-4626-8470-566aa010ba48-kube-api-access-qg2j7\") pod \"volume-data-source-validator-86695b7cc7-v9xsz\" (UID: \"08374dad-0ebd-4626-8470-566aa010ba48\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-v9xsz"
Sep 12 00:44:38.020403 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:38.020180    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls podName:ae656cec-be27-43b5-8123-f2bb132e6de3 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:38.52015876 +0000 UTC m=+83.080610424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kkwg8" (UID: "ae656cec-be27-43b5-8123-f2bb132e6de3") : secret "samples-operator-tls" not found
Sep 12 00:44:38.020708 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.020682    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a58e1e17-be8b-47d5-bdee-cd38e5554c44-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5pnzt\" (UID: \"a58e1e17-be8b-47d5-bdee-cd38e5554c44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt"
Sep 12 00:44:38.022535 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.022510    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a58e1e17-be8b-47d5-bdee-cd38e5554c44-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5pnzt\" (UID: \"a58e1e17-be8b-47d5-bdee-cd38e5554c44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt"
Sep 12 00:44:38.038024 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.037966    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-26rk9"]
Sep 12 00:44:38.042076 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.042051    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.042605 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.042578    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8stl5\" (UniqueName: \"kubernetes.io/projected/a58e1e17-be8b-47d5-bdee-cd38e5554c44-kube-api-access-8stl5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5pnzt\" (UID: \"a58e1e17-be8b-47d5-bdee-cd38e5554c44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt"
Sep 12 00:44:38.042605 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.042594    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6rx9\" (UniqueName: \"kubernetes.io/projected/ae656cec-be27-43b5-8123-f2bb132e6de3-kube-api-access-x6rx9\") pod \"cluster-samples-operator-86499c89c4-kkwg8\" (UID: \"ae656cec-be27-43b5-8123-f2bb132e6de3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"
Sep 12 00:44:38.043606 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.043586    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg2j7\" (UniqueName: \"kubernetes.io/projected/08374dad-0ebd-4626-8470-566aa010ba48-kube-api-access-qg2j7\") pod \"volume-data-source-validator-86695b7cc7-v9xsz\" (UID: \"08374dad-0ebd-4626-8470-566aa010ba48\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-v9xsz"
Sep 12 00:44:38.045487 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.045472    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 12 00:44:38.046264 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.046241    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 12 00:44:38.046415 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.046398    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 12 00:44:38.046801 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.046779    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 12 00:44:38.047364 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.047347    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ckxkz\""
Sep 12 00:44:38.052218 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.052197    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 12 00:44:38.060051 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.060032    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-26rk9"]
Sep 12 00:44:38.096315 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.096285    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-gbjdw"]
Sep 12 00:44:38.098587 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.098568    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz"
Sep 12 00:44:38.099552 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.099524    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:38.103161 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.103145    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 12 00:44:38.103834 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.103812    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 12 00:44:38.103909 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.103888    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 12 00:44:38.106056 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.106027    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gd22k\""
Sep 12 00:44:38.106318 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.106301    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:44:38.112298 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.112195    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-gbjdw"]
Sep 12 00:44:38.112805 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.112787    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 12 00:44:38.121201 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.121179    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aec6dfd1-8c25-4904-9234-23f02c32ea46-snapshots\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.121295 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.121227    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09725ba5-1c2a-4a09-9901-65dcf94d47d4-serving-cert\") pod \"console-operator-696884fb9c-gbjdw\" (UID: \"09725ba5-1c2a-4a09-9901-65dcf94d47d4\") " pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:38.121295 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.121251    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09725ba5-1c2a-4a09-9901-65dcf94d47d4-trusted-ca\") pod \"console-operator-696884fb9c-gbjdw\" (UID: \"09725ba5-1c2a-4a09-9901-65dcf94d47d4\") " pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:38.121295 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.121279    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09725ba5-1c2a-4a09-9901-65dcf94d47d4-config\") pod \"console-operator-696884fb9c-gbjdw\" (UID: \"09725ba5-1c2a-4a09-9901-65dcf94d47d4\") " pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:38.121295 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.121293    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aec6dfd1-8c25-4904-9234-23f02c32ea46-tmp\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.121436 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.121316    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec6dfd1-8c25-4904-9234-23f02c32ea46-service-ca-bundle\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.121436 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.121353    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aec6dfd1-8c25-4904-9234-23f02c32ea46-serving-cert\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.121436 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.121373    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz6dz\" (UniqueName: \"kubernetes.io/projected/aec6dfd1-8c25-4904-9234-23f02c32ea46-kube-api-access-xz6dz\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.121436 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.121404    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtfst\" (UniqueName: \"kubernetes.io/projected/09725ba5-1c2a-4a09-9901-65dcf94d47d4-kube-api-access-jtfst\") pod \"console-operator-696884fb9c-gbjdw\" (UID: \"09725ba5-1c2a-4a09-9901-65dcf94d47d4\") " pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:38.121563 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.121454    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec6dfd1-8c25-4904-9234-23f02c32ea46-trusted-ca-bundle\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.205305 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.205267    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-v9xsz"
Sep 12 00:44:38.213965 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.213920    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt"
Sep 12 00:44:38.214509 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.214487    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:38.221911 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.221888    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aec6dfd1-8c25-4904-9234-23f02c32ea46-snapshots\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.222035 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.221941    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09725ba5-1c2a-4a09-9901-65dcf94d47d4-serving-cert\") pod \"console-operator-696884fb9c-gbjdw\" (UID: \"09725ba5-1c2a-4a09-9901-65dcf94d47d4\") " pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:38.222035 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.221985    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09725ba5-1c2a-4a09-9901-65dcf94d47d4-trusted-ca\") pod \"console-operator-696884fb9c-gbjdw\" (UID: \"09725ba5-1c2a-4a09-9901-65dcf94d47d4\") " pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:38.222035 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.222021    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09725ba5-1c2a-4a09-9901-65dcf94d47d4-config\") pod \"console-operator-696884fb9c-gbjdw\" (UID: \"09725ba5-1c2a-4a09-9901-65dcf94d47d4\") " pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:38.222203 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.222044    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aec6dfd1-8c25-4904-9234-23f02c32ea46-tmp\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.222203 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.222067    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec6dfd1-8c25-4904-9234-23f02c32ea46-service-ca-bundle\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.222203 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.222097    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aec6dfd1-8c25-4904-9234-23f02c32ea46-serving-cert\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.222203 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.222119    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz6dz\" (UniqueName: \"kubernetes.io/projected/aec6dfd1-8c25-4904-9234-23f02c32ea46-kube-api-access-xz6dz\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.222203 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.222155    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtfst\" (UniqueName: \"kubernetes.io/projected/09725ba5-1c2a-4a09-9901-65dcf94d47d4-kube-api-access-jtfst\") pod \"console-operator-696884fb9c-gbjdw\" (UID: \"09725ba5-1c2a-4a09-9901-65dcf94d47d4\") " pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:38.222203 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.222203    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec6dfd1-8c25-4904-9234-23f02c32ea46-trusted-ca-bundle\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.222612 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.222562    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aec6dfd1-8c25-4904-9234-23f02c32ea46-tmp\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.222668 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.222627    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aec6dfd1-8c25-4904-9234-23f02c32ea46-snapshots\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.223142 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.222928    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09725ba5-1c2a-4a09-9901-65dcf94d47d4-trusted-ca\") pod \"console-operator-696884fb9c-gbjdw\" (UID: \"09725ba5-1c2a-4a09-9901-65dcf94d47d4\") " pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:38.223142 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.223028    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec6dfd1-8c25-4904-9234-23f02c32ea46-service-ca-bundle\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.223335 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.223310    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec6dfd1-8c25-4904-9234-23f02c32ea46-trusted-ca-bundle\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.223878 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.223853    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09725ba5-1c2a-4a09-9901-65dcf94d47d4-config\") pod \"console-operator-696884fb9c-gbjdw\" (UID: \"09725ba5-1c2a-4a09-9901-65dcf94d47d4\") " pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:38.224645 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.224628    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09725ba5-1c2a-4a09-9901-65dcf94d47d4-serving-cert\") pod \"console-operator-696884fb9c-gbjdw\" (UID: \"09725ba5-1c2a-4a09-9901-65dcf94d47d4\") " pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:38.225328 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.225299    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aec6dfd1-8c25-4904-9234-23f02c32ea46-serving-cert\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.231569 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.231543    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtfst\" (UniqueName: \"kubernetes.io/projected/09725ba5-1c2a-4a09-9901-65dcf94d47d4-kube-api-access-jtfst\") pod \"console-operator-696884fb9c-gbjdw\" (UID: \"09725ba5-1c2a-4a09-9901-65dcf94d47d4\") " pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:38.232196 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.232174    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz6dz\" (UniqueName: \"kubernetes.io/projected/aec6dfd1-8c25-4904-9234-23f02c32ea46-kube-api-access-xz6dz\") pod \"insights-operator-5844b856db-26rk9\" (UID: \"aec6dfd1-8c25-4904-9234-23f02c32ea46\") " pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.315791 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.315701    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:38.360000 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.359974    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-26rk9"
Sep 12 00:44:38.368347 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.368325    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz"]
Sep 12 00:44:38.372222 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:44:38.372194    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b0b2b4_0df2_410f_89d0_705dd447dc8a.slice/crio-14c18acd69ca71126c0b223a8078de0550885c71daca5214410a0e2fa397dfaf WatchSource:0}: Error finding container 14c18acd69ca71126c0b223a8078de0550885c71daca5214410a0e2fa397dfaf: Status 404 returned error can't find the container with id 14c18acd69ca71126c0b223a8078de0550885c71daca5214410a0e2fa397dfaf
Sep 12 00:44:38.410908 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.410736    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:38.412477 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.412444    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz" event={"ID":"48b0b2b4-0df2-410f-89d0-705dd447dc8a","Type":"ContainerStarted","Data":"14c18acd69ca71126c0b223a8078de0550885c71daca5214410a0e2fa397dfaf"}
Sep 12 00:44:38.481215 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.481186    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-v9xsz"]
Sep 12 00:44:38.483888 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:44:38.483861    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08374dad_0ebd_4626_8470_566aa010ba48.slice/crio-5d9d95052c91f1710a539d01c267aa592fccbc3613390896ea5cac8b234d75a0 WatchSource:0}: Error finding container 5d9d95052c91f1710a539d01c267aa592fccbc3613390896ea5cac8b234d75a0: Status 404 returned error can't find the container with id 5d9d95052c91f1710a539d01c267aa592fccbc3613390896ea5cac8b234d75a0
Sep 12 00:44:38.487575 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.487493    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt"]
Sep 12 00:44:38.490241 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:44:38.490220    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58e1e17_be8b_47d5_bdee_cd38e5554c44.slice/crio-980d9cec8a1342bfd71631a6b6051238441447dbbd3773942acb34b9359537f0 WatchSource:0}: Error finding container 980d9cec8a1342bfd71631a6b6051238441447dbbd3773942acb34b9359537f0: Status 404 returned error can't find the container with id 980d9cec8a1342bfd71631a6b6051238441447dbbd3773942acb34b9359537f0
Sep 12 00:44:38.523970 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.523921    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kkwg8\" (UID: \"ae656cec-be27-43b5-8123-f2bb132e6de3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"
Sep 12 00:44:38.524108 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:38.524065    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 12 00:44:38.524155 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:38.524122    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls podName:ae656cec-be27-43b5-8123-f2bb132e6de3 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:39.524107103 +0000 UTC m=+84.084558749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kkwg8" (UID: "ae656cec-be27-43b5-8123-f2bb132e6de3") : secret "samples-operator-tls" not found
Sep 12 00:44:38.628658 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.628587    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-26rk9"]
Sep 12 00:44:38.631483 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:44:38.631458    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec6dfd1_8c25_4904_9234_23f02c32ea46.slice/crio-eb0b9e215e22b156f63a864313c4b8fd1900f675247e85d51056d02c498ab356 WatchSource:0}: Error finding container eb0b9e215e22b156f63a864313c4b8fd1900f675247e85d51056d02c498ab356: Status 404 returned error can't find the container with id eb0b9e215e22b156f63a864313c4b8fd1900f675247e85d51056d02c498ab356
Sep 12 00:44:38.680648 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:38.680616    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-gbjdw"]
Sep 12 00:44:38.683972 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:44:38.683918    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09725ba5_1c2a_4a09_9901_65dcf94d47d4.slice/crio-fe8275fb25b61b898c42f766073cc4826280d31fca29d44babcc69c2f8f4dc42 WatchSource:0}: Error finding container fe8275fb25b61b898c42f766073cc4826280d31fca29d44babcc69c2f8f4dc42: Status 404 returned error can't find the container with id fe8275fb25b61b898c42f766073cc4826280d31fca29d44babcc69c2f8f4dc42
Sep 12 00:44:39.418876 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:39.418794    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-v9xsz" event={"ID":"08374dad-0ebd-4626-8470-566aa010ba48","Type":"ContainerStarted","Data":"5d9d95052c91f1710a539d01c267aa592fccbc3613390896ea5cac8b234d75a0"}
Sep 12 00:44:39.421231 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:39.421158    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-gbjdw" event={"ID":"09725ba5-1c2a-4a09-9901-65dcf94d47d4","Type":"ContainerStarted","Data":"fe8275fb25b61b898c42f766073cc4826280d31fca29d44babcc69c2f8f4dc42"}
Sep 12 00:44:39.423925 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:39.423860    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-26rk9" event={"ID":"aec6dfd1-8c25-4904-9234-23f02c32ea46","Type":"ContainerStarted","Data":"eb0b9e215e22b156f63a864313c4b8fd1900f675247e85d51056d02c498ab356"}
Sep 12 00:44:39.425262 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:39.425203    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt" event={"ID":"a58e1e17-be8b-47d5-bdee-cd38e5554c44","Type":"ContainerStarted","Data":"980d9cec8a1342bfd71631a6b6051238441447dbbd3773942acb34b9359537f0"}
Sep 12 00:44:39.533873 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:39.533276    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kkwg8\" (UID: \"ae656cec-be27-43b5-8123-f2bb132e6de3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"
Sep 12 00:44:39.533873 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:39.533441    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 12 00:44:39.533873 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:39.533508    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls podName:ae656cec-be27-43b5-8123-f2bb132e6de3 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:41.533487668 +0000 UTC m=+86.093939333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kkwg8" (UID: "ae656cec-be27-43b5-8123-f2bb132e6de3") : secret "samples-operator-tls" not found
Sep 12 00:44:40.429589 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:40.429552    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-v9xsz" event={"ID":"08374dad-0ebd-4626-8470-566aa010ba48","Type":"ContainerStarted","Data":"01dd99b771fc78c1e36adb9e2c21a19197c3bea3e4017bad4787a769681aa952"}
Sep 12 00:44:40.447866 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:40.447801    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-v9xsz" podStartSLOduration=2.157028814 podStartE2EDuration="3.447779143s" totalImagesPullingTime="1.290750329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:40.446671658 +0000 UTC m=+85.007123351" watchObservedRunningTime="2026-09-12 00:44:40.447779143 +0000 UTC m=+85.008230814"
Sep 12 00:44:40.578850 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:40.578806    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:41.550014 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:41.549969    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kkwg8\" (UID: \"ae656cec-be27-43b5-8123-f2bb132e6de3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"
Sep 12 00:44:41.550575 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:41.550158    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 12 00:44:41.550575 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:41.550228    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls podName:ae656cec-be27-43b5-8123-f2bb132e6de3 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:45.550208631 +0000 UTC m=+90.110660295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kkwg8" (UID: "ae656cec-be27-43b5-8123-f2bb132e6de3") : secret "samples-operator-tls" not found
Sep 12 00:44:41.846848 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:41.846767    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"]
Sep 12 00:44:41.851041 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:41.851015    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"
Sep 12 00:44:41.853779 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:41.853753    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 12 00:44:41.853779 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:41.853771    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-zjdgf\""
Sep 12 00:44:41.853993 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:41.853776    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 12 00:44:41.860203 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:41.860173    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"]
Sep 12 00:44:41.955715 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:41.955682    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/653056a7-146e-44bd-bd1d-41cd75537411-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-svchm\" (UID: \"653056a7-146e-44bd-bd1d-41cd75537411\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"
Sep 12 00:44:41.955882 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:41.955722    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-svchm\" (UID: \"653056a7-146e-44bd-bd1d-41cd75537411\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"
Sep 12 00:44:42.056854 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:42.056817    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/653056a7-146e-44bd-bd1d-41cd75537411-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-svchm\" (UID: \"653056a7-146e-44bd-bd1d-41cd75537411\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"
Sep 12 00:44:42.056854 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:42.056855    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-svchm\" (UID: \"653056a7-146e-44bd-bd1d-41cd75537411\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"
Sep 12 00:44:42.057124 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:42.056977    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 00:44:42.057124 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:42.057034    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert podName:653056a7-146e-44bd-bd1d-41cd75537411 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:42.557018857 +0000 UTC m=+87.117470503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-svchm" (UID: "653056a7-146e-44bd-bd1d-41cd75537411") : secret "networking-console-plugin-cert" not found
Sep 12 00:44:42.057586 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:42.057560    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/653056a7-146e-44bd-bd1d-41cd75537411-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-svchm\" (UID: \"653056a7-146e-44bd-bd1d-41cd75537411\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"
Sep 12 00:44:42.437497 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:42.437469    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/0.log"
Sep 12 00:44:42.437685 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:42.437512    2567 generic.go:356] "Generic (PLEG): container finished" podID="09725ba5-1c2a-4a09-9901-65dcf94d47d4" containerID="1933ee0ecb84dae0e87da9f4746b28238b8a02c921fd05e386a2eb58f761e1d3" exitCode=255
Sep 12 00:44:42.437685 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:42.437569    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-gbjdw" event={"ID":"09725ba5-1c2a-4a09-9901-65dcf94d47d4","Type":"ContainerDied","Data":"1933ee0ecb84dae0e87da9f4746b28238b8a02c921fd05e386a2eb58f761e1d3"}
Sep 12 00:44:42.437920 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:42.437851    2567 scope.go:117] "RemoveContainer" containerID="1933ee0ecb84dae0e87da9f4746b28238b8a02c921fd05e386a2eb58f761e1d3"
Sep 12 00:44:42.439392 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:42.439341    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-26rk9" event={"ID":"aec6dfd1-8c25-4904-9234-23f02c32ea46","Type":"ContainerStarted","Data":"dac3444d9620da266adc426ae75d735a7ab342de7520324466cb32d978a43e1b"}
Sep 12 00:44:42.441124 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:42.441095    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt" event={"ID":"a58e1e17-be8b-47d5-bdee-cd38e5554c44","Type":"ContainerStarted","Data":"f648122c50dbee3e03c4a75a587852ae5ee7074eb19a4a12286762fd1299c6ec"}
Sep 12 00:44:42.442895 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:42.442872    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz" event={"ID":"48b0b2b4-0df2-410f-89d0-705dd447dc8a","Type":"ContainerStarted","Data":"eec5d059bebdbf6b4ec92f421d8382abff95863f808be5810e488aa89501e186"}
Sep 12 00:44:42.477082 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:42.476475    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-26rk9" podStartSLOduration=1.9500345989999999 podStartE2EDuration="5.476455288s" totalImagesPullingTime="3.526420689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:42.476393414 +0000 UTC m=+87.036845100" watchObservedRunningTime="2026-09-12 00:44:42.476455288 +0000 UTC m=+87.036906955"
Sep 12 00:44:42.497515 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:42.497455    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt" podStartSLOduration=1.838109431 podStartE2EDuration="5.497438682s" totalImagesPullingTime="3.659329251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:42.497360419 +0000 UTC m=+87.057812082" watchObservedRunningTime="2026-09-12 00:44:42.497438682 +0000 UTC m=+87.057890352"
Sep 12 00:44:42.524394 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:42.524338    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz" podStartSLOduration=1.744566297 podStartE2EDuration="5.524318651s" totalImagesPullingTime="3.779752354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:42.52381676 +0000 UTC m=+87.084268430" watchObservedRunningTime="2026-09-12 00:44:42.524318651 +0000 UTC m=+87.084770311"
Sep 12 00:44:42.561076 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:42.560934    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-svchm\" (UID: \"653056a7-146e-44bd-bd1d-41cd75537411\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"
Sep 12 00:44:42.561501 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:42.561089    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 00:44:42.561501 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:42.561173    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert podName:653056a7-146e-44bd-bd1d-41cd75537411 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:43.561150777 +0000 UTC m=+88.121602437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-svchm" (UID: "653056a7-146e-44bd-bd1d-41cd75537411") : secret "networking-console-plugin-cert" not found
Sep 12 00:44:43.031494 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.031460    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:43.061650 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.061619    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-hdt7r"]
Sep 12 00:44:43.065041 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.065017    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hdt7r"
Sep 12 00:44:43.067773 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.067538    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nl6zw\""
Sep 12 00:44:43.068667 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.068646    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 12 00:44:43.068841 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.068817    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 12 00:44:43.075653 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.075632    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-hdt7r"]
Sep 12 00:44:43.123473 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.123439    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:43.165696 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.165615    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8zvr\" (UniqueName: \"kubernetes.io/projected/24e688c4-27f7-4830-969b-af15d8962683-kube-api-access-g8zvr\") pod \"migrator-8b4d697c5-hdt7r\" (UID: \"24e688c4-27f7-4830-969b-af15d8962683\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hdt7r"
Sep 12 00:44:43.187167 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.187131    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:43.266240 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.266204    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8zvr\" (UniqueName: \"kubernetes.io/projected/24e688c4-27f7-4830-969b-af15d8962683-kube-api-access-g8zvr\") pod \"migrator-8b4d697c5-hdt7r\" (UID: \"24e688c4-27f7-4830-969b-af15d8962683\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hdt7r"
Sep 12 00:44:43.275361 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.275302    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8zvr\" (UniqueName: \"kubernetes.io/projected/24e688c4-27f7-4830-969b-af15d8962683-kube-api-access-g8zvr\") pod \"migrator-8b4d697c5-hdt7r\" (UID: \"24e688c4-27f7-4830-969b-af15d8962683\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hdt7r"
Sep 12 00:44:43.351334 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.351247    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:43.351558 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.351538    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rp657"]
Sep 12 00:44:43.355017 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.354995    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.357697 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.357672    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 12 00:44:43.357807 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.357713    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 12 00:44:43.357807 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.357738    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d5lqq\""
Sep 12 00:44:43.365583 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.365562    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rp657"]
Sep 12 00:44:43.367513 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.367489    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19b79c5c-195d-436e-8785-90cc7e91d8cb-data-volume\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.367613 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.367525    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.367691 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.367667    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19b79c5c-195d-436e-8785-90cc7e91d8cb-crio-socket\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.367815 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.367698    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.367815 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.367734    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6fb2\" (UniqueName: \"kubernetes.io/projected/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-api-access-l6fb2\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.377484 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.377461    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hdt7r"
Sep 12 00:44:43.419851 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.419818    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:43.448927 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.448903    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 00:44:43.449344 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.449329    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/0.log"
Sep 12 00:44:43.449405 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.449359    2567 generic.go:356] "Generic (PLEG): container finished" podID="09725ba5-1c2a-4a09-9901-65dcf94d47d4" containerID="4b28d20ef63a3ef00a4b939fc1c685f45ca430e69f6a69ec55778f69bc25e72b" exitCode=255
Sep 12 00:44:43.450025 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.450006    2567 scope.go:117] "RemoveContainer" containerID="4b28d20ef63a3ef00a4b939fc1c685f45ca430e69f6a69ec55778f69bc25e72b"
Sep 12 00:44:43.450426 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.450147    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-gbjdw" event={"ID":"09725ba5-1c2a-4a09-9901-65dcf94d47d4","Type":"ContainerDied","Data":"4b28d20ef63a3ef00a4b939fc1c685f45ca430e69f6a69ec55778f69bc25e72b"}
Sep 12 00:44:43.450426 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:43.450176    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-gbjdw_openshift-console-operator(09725ba5-1c2a-4a09-9901-65dcf94d47d4)\"" pod="openshift-console-operator/console-operator-696884fb9c-gbjdw" podUID="09725ba5-1c2a-4a09-9901-65dcf94d47d4"
Sep 12 00:44:43.450426 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.450206    2567 scope.go:117] "RemoveContainer" containerID="1933ee0ecb84dae0e87da9f4746b28238b8a02c921fd05e386a2eb58f761e1d3"
Sep 12 00:44:43.468272 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.468245    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19b79c5c-195d-436e-8785-90cc7e91d8cb-data-volume\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.468399 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.468275    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.468399 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:43.468360    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:43.468518 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:43.468403    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls podName:19b79c5c-195d-436e-8785-90cc7e91d8cb nodeName:}" failed. No retries permitted until 2026-09-12 00:44:43.968390224 +0000 UTC m=+88.528841883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rp657" (UID: "19b79c5c-195d-436e-8785-90cc7e91d8cb") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:43.469256 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.468858    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19b79c5c-195d-436e-8785-90cc7e91d8cb-data-volume\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.469256 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.468884    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19b79c5c-195d-436e-8785-90cc7e91d8cb-crio-socket\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.469256 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.468917    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.469256 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.468981    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19b79c5c-195d-436e-8785-90cc7e91d8cb-crio-socket\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.469256 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.468994    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6fb2\" (UniqueName: \"kubernetes.io/projected/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-api-access-l6fb2\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.469256 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:43.469213    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:44:43.469256 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:43.469257    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm podName:19b79c5c-195d-436e-8785-90cc7e91d8cb nodeName:}" failed. No retries permitted until 2026-09-12 00:44:43.969245139 +0000 UTC m=+88.529696824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rp657" (UID: "19b79c5c-195d-436e-8785-90cc7e91d8cb") : configmap "kube-rbac-proxy" not found
Sep 12 00:44:43.479129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.479110    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6fb2\" (UniqueName: \"kubernetes.io/projected/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-api-access-l6fb2\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.569934 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.569890    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-svchm\" (UID: \"653056a7-146e-44bd-bd1d-41cd75537411\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"
Sep 12 00:44:43.570326 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:43.570054    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 00:44:43.570326 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:43.570133    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert podName:653056a7-146e-44bd-bd1d-41cd75537411 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:45.570112468 +0000 UTC m=+90.130564114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-svchm" (UID: "653056a7-146e-44bd-bd1d-41cd75537411") : secret "networking-console-plugin-cert" not found
Sep 12 00:44:43.651216 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.651032    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-hdt7r"]
Sep 12 00:44:43.653596 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:44:43.653563    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e688c4_27f7_4830_969b_af15d8962683.slice/crio-a318a3199078cffba70ff856d05c30e1ada0235c79bd34e812aea9d3e484c721 WatchSource:0}: Error finding container a318a3199078cffba70ff856d05c30e1ada0235c79bd34e812aea9d3e484c721: Status 404 returned error can't find the container with id a318a3199078cffba70ff856d05c30e1ada0235c79bd34e812aea9d3e484c721
Sep 12 00:44:43.973193 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.973159    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.973368 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:43.973254    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:43.973368 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:43.973304    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:43.973368 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:43.973358    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:44:43.973488 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:43.973370    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls podName:19b79c5c-195d-436e-8785-90cc7e91d8cb nodeName:}" failed. No retries permitted until 2026-09-12 00:44:44.973354303 +0000 UTC m=+89.533805954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rp657" (UID: "19b79c5c-195d-436e-8785-90cc7e91d8cb") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:43.973488 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:43.973402    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm podName:19b79c5c-195d-436e-8785-90cc7e91d8cb nodeName:}" failed. No retries permitted until 2026-09-12 00:44:44.973390365 +0000 UTC m=+89.533842011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rp657" (UID: "19b79c5c-195d-436e-8785-90cc7e91d8cb") : configmap "kube-rbac-proxy" not found
Sep 12 00:44:44.420091 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:44.420050    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:44.455023 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:44.454996    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 00:44:44.455407 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:44.455385    2567 scope.go:117] "RemoveContainer" containerID="4b28d20ef63a3ef00a4b939fc1c685f45ca430e69f6a69ec55778f69bc25e72b"
Sep 12 00:44:44.455566 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:44.455547    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-gbjdw_openshift-console-operator(09725ba5-1c2a-4a09-9901-65dcf94d47d4)\"" pod="openshift-console-operator/console-operator-696884fb9c-gbjdw" podUID="09725ba5-1c2a-4a09-9901-65dcf94d47d4"
Sep 12 00:44:44.456551 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:44.456528    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hdt7r" event={"ID":"24e688c4-27f7-4830-969b-af15d8962683","Type":"ContainerStarted","Data":"a318a3199078cffba70ff856d05c30e1ada0235c79bd34e812aea9d3e484c721"}
Sep 12 00:44:44.483359 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:44.483331    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:44.847886 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:44.847857    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:44.950252 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:44.950150    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:44.982264 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:44.982234    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:44.982411 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:44.982316    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:44.982411 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:44.982382    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:44.982535 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:44.982419    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:44:44.982535 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:44.982443    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls podName:19b79c5c-195d-436e-8785-90cc7e91d8cb nodeName:}" failed. No retries permitted until 2026-09-12 00:44:46.982428635 +0000 UTC m=+91.542880281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rp657" (UID: "19b79c5c-195d-436e-8785-90cc7e91d8cb") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:44.982535 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:44.982481    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm podName:19b79c5c-195d-436e-8785-90cc7e91d8cb nodeName:}" failed. No retries permitted until 2026-09-12 00:44:46.982465514 +0000 UTC m=+91.542917182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rp657" (UID: "19b79c5c-195d-436e-8785-90cc7e91d8cb") : configmap "kube-rbac-proxy" not found
Sep 12 00:44:45.129419 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:45.129386    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:45.230281 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:45.230193    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:45.465449 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:45.465416    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hdt7r" event={"ID":"24e688c4-27f7-4830-969b-af15d8962683","Type":"ContainerStarted","Data":"ef782bcc7fe3b50d179ad7b97efbe74defb30febe95b823c96c802242ee8e590"}
Sep 12 00:44:45.465655 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:45.465508    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hdt7r" event={"ID":"24e688c4-27f7-4830-969b-af15d8962683","Type":"ContainerStarted","Data":"16009d407a7311567d9a1b4e61028dd9d4af6e3441e2f0146c002daa65563316"}
Sep 12 00:44:45.482347 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:45.482224    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hdt7r" podStartSLOduration=1.494981662 podStartE2EDuration="2.482208842s" totalImagesPullingTime="987.22718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:45.482003159 +0000 UTC m=+90.042454843" watchObservedRunningTime="2026-09-12 00:44:45.482208842 +0000 UTC m=+90.042660566"
Sep 12 00:44:45.567852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:45.567806    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:45.586940 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:45.586905    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-svchm\" (UID: \"653056a7-146e-44bd-bd1d-41cd75537411\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"
Sep 12 00:44:45.587121 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:45.587025    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kkwg8\" (UID: \"ae656cec-be27-43b5-8123-f2bb132e6de3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"
Sep 12 00:44:45.587121 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:45.587045    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 00:44:45.587121 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:45.587103    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert podName:653056a7-146e-44bd-bd1d-41cd75537411 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:49.587088419 +0000 UTC m=+94.147540066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-svchm" (UID: "653056a7-146e-44bd-bd1d-41cd75537411") : secret "networking-console-plugin-cert" not found
Sep 12 00:44:45.587254 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:45.587157    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 12 00:44:45.587254 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:45.587210    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls podName:ae656cec-be27-43b5-8123-f2bb132e6de3 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:53.587195537 +0000 UTC m=+98.147647183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kkwg8" (UID: "ae656cec-be27-43b5-8123-f2bb132e6de3") : secret "samples-operator-tls" not found
Sep 12 00:44:45.668183 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:45.668148    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:46.099399 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.099375    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cbjgx_2337c617-d3c5-417f-b2c8-4c8328a04a58/dns-node-resolver/0.log"
Sep 12 00:44:46.722161 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.722125    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-ffcgb"]
Sep 12 00:44:46.725541 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.725523    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-ffcgb"
Sep 12 00:44:46.728274 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.728250    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-86sjf\""
Sep 12 00:44:46.728401 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.728251    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 12 00:44:46.729117 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.729092    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 12 00:44:46.729224 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.729131    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 12 00:44:46.729224 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.729170    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 12 00:44:46.733187 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.733163    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-ffcgb"]
Sep 12 00:44:46.771399 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.771366    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:46.796817 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.796789    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl8fs\" (UniqueName: \"kubernetes.io/projected/01e0b189-cf02-4c50-842a-4299f174aeb4-kube-api-access-dl8fs\") pod \"service-ca-568869fdc5-ffcgb\" (UID: \"01e0b189-cf02-4c50-842a-4299f174aeb4\") " pod="openshift-service-ca/service-ca-568869fdc5-ffcgb"
Sep 12 00:44:46.796999 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.796871    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01e0b189-cf02-4c50-842a-4299f174aeb4-signing-key\") pod \"service-ca-568869fdc5-ffcgb\" (UID: \"01e0b189-cf02-4c50-842a-4299f174aeb4\") " pod="openshift-service-ca/service-ca-568869fdc5-ffcgb"
Sep 12 00:44:46.796999 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.796915    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01e0b189-cf02-4c50-842a-4299f174aeb4-signing-cabundle\") pod \"service-ca-568869fdc5-ffcgb\" (UID: \"01e0b189-cf02-4c50-842a-4299f174aeb4\") " pod="openshift-service-ca/service-ca-568869fdc5-ffcgb"
Sep 12 00:44:46.827395 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.827367    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:46.897964 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.897916    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01e0b189-cf02-4c50-842a-4299f174aeb4-signing-key\") pod \"service-ca-568869fdc5-ffcgb\" (UID: \"01e0b189-cf02-4c50-842a-4299f174aeb4\") " pod="openshift-service-ca/service-ca-568869fdc5-ffcgb"
Sep 12 00:44:46.898151 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.897979    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01e0b189-cf02-4c50-842a-4299f174aeb4-signing-cabundle\") pod \"service-ca-568869fdc5-ffcgb\" (UID: \"01e0b189-cf02-4c50-842a-4299f174aeb4\") " pod="openshift-service-ca/service-ca-568869fdc5-ffcgb"
Sep 12 00:44:46.898151 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.898064    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8fs\" (UniqueName: \"kubernetes.io/projected/01e0b189-cf02-4c50-842a-4299f174aeb4-kube-api-access-dl8fs\") pod \"service-ca-568869fdc5-ffcgb\" (UID: \"01e0b189-cf02-4c50-842a-4299f174aeb4\") " pod="openshift-service-ca/service-ca-568869fdc5-ffcgb"
Sep 12 00:44:46.898743 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.898725    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01e0b189-cf02-4c50-842a-4299f174aeb4-signing-cabundle\") pod \"service-ca-568869fdc5-ffcgb\" (UID: \"01e0b189-cf02-4c50-842a-4299f174aeb4\") " pod="openshift-service-ca/service-ca-568869fdc5-ffcgb"
Sep 12 00:44:46.900491 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.900473    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01e0b189-cf02-4c50-842a-4299f174aeb4-signing-key\") pod \"service-ca-568869fdc5-ffcgb\" (UID: \"01e0b189-cf02-4c50-842a-4299f174aeb4\") " pod="openshift-service-ca/service-ca-568869fdc5-ffcgb"
Sep 12 00:44:46.907423 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.907402    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8fs\" (UniqueName: \"kubernetes.io/projected/01e0b189-cf02-4c50-842a-4299f174aeb4-kube-api-access-dl8fs\") pod \"service-ca-568869fdc5-ffcgb\" (UID: \"01e0b189-cf02-4c50-842a-4299f174aeb4\") " pod="openshift-service-ca/service-ca-568869fdc5-ffcgb"
Sep 12 00:44:46.999331 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.999235    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:46.999331 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:46.999316    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:46.999514 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:46.999365    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:44:46.999514 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:46.999423    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:46.999514 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:46.999430    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm podName:19b79c5c-195d-436e-8785-90cc7e91d8cb nodeName:}" failed. No retries permitted until 2026-09-12 00:44:50.999415108 +0000 UTC m=+95.559866753 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rp657" (UID: "19b79c5c-195d-436e-8785-90cc7e91d8cb") : configmap "kube-rbac-proxy" not found
Sep 12 00:44:46.999514 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:46.999469    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls podName:19b79c5c-195d-436e-8785-90cc7e91d8cb nodeName:}" failed. No retries permitted until 2026-09-12 00:44:50.99945754 +0000 UTC m=+95.559909203 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rp657" (UID: "19b79c5c-195d-436e-8785-90cc7e91d8cb") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:47.035264 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:47.035233    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-ffcgb"
Sep 12 00:44:47.299703 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:47.299632    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mvtjp_5bf0d680-e19c-4892-8652-58f49ed9d3b3/node-ca/0.log"
Sep 12 00:44:47.311087 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:47.311044    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-ffcgb"]
Sep 12 00:44:47.315267 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:44:47.315237    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e0b189_cf02_4c50_842a_4299f174aeb4.slice/crio-232f3dc86b762f4d9b3f051df0e3e8ee284c4ba87cf0895420a84e9a82fdd211 WatchSource:0}: Error finding container 232f3dc86b762f4d9b3f051df0e3e8ee284c4ba87cf0895420a84e9a82fdd211: Status 404 returned error can't find the container with id 232f3dc86b762f4d9b3f051df0e3e8ee284c4ba87cf0895420a84e9a82fdd211
Sep 12 00:44:47.478130 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:47.478093    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-ffcgb" event={"ID":"01e0b189-cf02-4c50-842a-4299f174aeb4","Type":"ContainerStarted","Data":"9f1ea37b2e9b1cb69f0e68c447e6b5f748757c0e7429b77edc6fe96bbac76449"}
Sep 12 00:44:47.478288 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:47.478141    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-ffcgb" event={"ID":"01e0b189-cf02-4c50-842a-4299f174aeb4","Type":"ContainerStarted","Data":"232f3dc86b762f4d9b3f051df0e3e8ee284c4ba87cf0895420a84e9a82fdd211"}
Sep 12 00:44:47.497159 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:47.497110    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-ffcgb" podStartSLOduration=1.497090853 podStartE2EDuration="1.497090853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:47.495631415 +0000 UTC m=+92.056083115" watchObservedRunningTime="2026-09-12 00:44:47.497090853 +0000 UTC m=+92.057542521"
Sep 12 00:44:47.698026 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:47.697943    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6z6bf_2ee67b56-1998-4545-9ceb-8597c6101dab/serve-healthcheck-canary/0.log"
Sep 12 00:44:47.744455 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:47.744419    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:48.298930 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:48.298902    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-hdt7r_24e688c4-27f7-4830-969b-af15d8962683/migrator/0.log"
Sep 12 00:44:48.410941 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:48.410903    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:48.411398 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:48.411002    2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:44:48.411931 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:48.411894    2567 scope.go:117] "RemoveContainer" containerID="4b28d20ef63a3ef00a4b939fc1c685f45ca430e69f6a69ec55778f69bc25e72b"
Sep 12 00:44:48.414468 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:48.412230    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-gbjdw_openshift-console-operator(09725ba5-1c2a-4a09-9901-65dcf94d47d4)\"" pod="openshift-console-operator/console-operator-696884fb9c-gbjdw" podUID="09725ba5-1c2a-4a09-9901-65dcf94d47d4"
Sep 12 00:44:48.497886 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:48.497857    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-hdt7r_24e688c4-27f7-4830-969b-af15d8962683/graceful-termination/0.log"
Sep 12 00:44:48.699011 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:48.698977    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-5pnzt_a58e1e17-be8b-47d5-bdee-cd38e5554c44/kube-storage-version-migrator-operator/0.log"
Sep 12 00:44:49.267006 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:49.266973    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:49.381281 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:49.381242    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:49.480801 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:49.480767    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:49.581737 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:49.581643    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:49.622068 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:49.622031    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-svchm\" (UID: \"653056a7-146e-44bd-bd1d-41cd75537411\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"
Sep 12 00:44:49.622354 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:49.622133    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 00:44:49.622467 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:49.622409    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert podName:653056a7-146e-44bd-bd1d-41cd75537411 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:57.622387044 +0000 UTC m=+102.182838707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-svchm" (UID: "653056a7-146e-44bd-bd1d-41cd75537411") : secret "networking-console-plugin-cert" not found
Sep 12 00:44:49.681775 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:49.681724    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:50.317359 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:50.317331    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:50.735024 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:50.734995    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:51.034663 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:51.034558    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:51.034663 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:51.034666    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:51.034848 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:51.034733    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:51.034848 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:51.034781    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:44:51.034848 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:51.034798    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls podName:19b79c5c-195d-436e-8785-90cc7e91d8cb nodeName:}" failed. No retries permitted until 2026-09-12 00:44:59.034780735 +0000 UTC m=+103.595232388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rp657" (UID: "19b79c5c-195d-436e-8785-90cc7e91d8cb") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:51.034848 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:44:51.034846    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm podName:19b79c5c-195d-436e-8785-90cc7e91d8cb nodeName:}" failed. No retries permitted until 2026-09-12 00:44:59.034832551 +0000 UTC m=+103.595284202 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rp657" (UID: "19b79c5c-195d-436e-8785-90cc7e91d8cb") : configmap "kube-rbac-proxy" not found
Sep 12 00:44:51.128738 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:51.128703    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:51.316853 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:51.316760    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:51.723718 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:51.723683    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:51.796823 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:51.796778    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:52.324093 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:52.324062    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:52.641207 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:52.641123    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:52.927701 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:52.927671    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:53.140561 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:53.140516    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:53.212137 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:53.212046    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:53.325968 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:53.325923    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:53.401556 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:53.401529    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:53.656542 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:53.656514    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kkwg8\" (UID: \"ae656cec-be27-43b5-8123-f2bb132e6de3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"
Sep 12 00:44:53.659122 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:53.659102    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae656cec-be27-43b5-8123-f2bb132e6de3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kkwg8\" (UID: \"ae656cec-be27-43b5-8123-f2bb132e6de3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"
Sep 12 00:44:53.719156 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:53.719127    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:53.819574 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:53.819543    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"
Sep 12 00:44:54.089359 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:54.089290    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8"]
Sep 12 00:44:54.261098 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:54.261059    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:44:54.263659 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:54.263631    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d07d13ab-1c44-499a-aa9f-ebed6bbe188e-metrics-tls\") pod \"dns-default-dsbc2\" (UID: \"d07d13ab-1c44-499a-aa9f-ebed6bbe188e\") " pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:44:54.269528 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:54.269505    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2gxdg\""
Sep 12 00:44:54.277139 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:54.277111    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:44:54.321874 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:54.321806    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:54.503197 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:54.503167    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8" event={"ID":"ae656cec-be27-43b5-8123-f2bb132e6de3","Type":"ContainerStarted","Data":"d24564fa0a42dcafd652560da1439380b815c19532d12441b08a6c6bcfc26a4d"}
Sep 12 00:44:54.562529 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:54.562484    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dsbc2"]
Sep 12 00:44:54.567411 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:44:54.567382    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07d13ab_1c44_499a_aa9f_ebed6bbe188e.slice/crio-9f13aabfe6dc3bbddcb9c6fa805780fdee18d01bb454e789470cc2af84b8d6ff WatchSource:0}: Error finding container 9f13aabfe6dc3bbddcb9c6fa805780fdee18d01bb454e789470cc2af84b8d6ff: Status 404 returned error can't find the container with id 9f13aabfe6dc3bbddcb9c6fa805780fdee18d01bb454e789470cc2af84b8d6ff
Sep 12 00:44:54.721582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:54.721505    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:55.118332 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:55.118294    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:55.510286 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:55.510247    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dsbc2" event={"ID":"d07d13ab-1c44-499a-aa9f-ebed6bbe188e","Type":"ContainerStarted","Data":"9f13aabfe6dc3bbddcb9c6fa805780fdee18d01bb454e789470cc2af84b8d6ff"}
Sep 12 00:44:55.521665 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:55.521573    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:55.621315 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:55.621253    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:55.721662 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:55.721625    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:56.114018 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:56.113909    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:56.191398 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:56.191355    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:56.514421 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:56.514397    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8" event={"ID":"ae656cec-be27-43b5-8123-f2bb132e6de3","Type":"ContainerStarted","Data":"bd9ea035f97ad97d4cf041f9dfbd5afd20f09c3f0fdc67a228a76b3584b94bde"}
Sep 12 00:44:56.514530 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:56.514448    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8" event={"ID":"ae656cec-be27-43b5-8123-f2bb132e6de3","Type":"ContainerStarted","Data":"597a0acf90690fb7127b4c97075253320ecf17138653099b45fc92f756d39656"}
Sep 12 00:44:56.530373 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:56.530303    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kkwg8" podStartSLOduration=17.825065199 podStartE2EDuration="19.530281535s" totalImagesPullingTime="1.705216336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:56.529426447 +0000 UTC m=+101.089878138" watchObservedRunningTime="2026-09-12 00:44:56.530281535 +0000 UTC m=+101.090733203"
Sep 12 00:44:56.910620 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:56.910570    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:57.118331 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:57.118297    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:57.386849 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:57.386777    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bdgdv"
Sep 12 00:44:57.514799 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:57.514758    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:57.519464 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:57.519436    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dsbc2" event={"ID":"d07d13ab-1c44-499a-aa9f-ebed6bbe188e","Type":"ContainerStarted","Data":"d3ea4a1d91411496b7e9bfc4f3d97b3605fad0771949c598477b070e66bc7773"}
Sep 12 00:44:57.519580 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:57.519497    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dsbc2" event={"ID":"d07d13ab-1c44-499a-aa9f-ebed6bbe188e","Type":"ContainerStarted","Data":"027eba5f2b0484230cc7a44d5d408cc38c97e910230fad0602aae2a468f8ec0e"}
Sep 12 00:44:57.536882 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:57.536833    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dsbc2" podStartSLOduration=65.597816302 podStartE2EDuration="1m7.536819394s" totalImagesPullingTime="1.939003092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:57.535490685 +0000 UTC m=+102.095942375" watchObservedRunningTime="2026-09-12 00:44:57.536819394 +0000 UTC m=+102.097271062"
Sep 12 00:44:57.689995 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:57.689939    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-svchm\" (UID: \"653056a7-146e-44bd-bd1d-41cd75537411\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"
Sep 12 00:44:57.692673 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:57.692643    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/653056a7-146e-44bd-bd1d-41cd75537411-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-svchm\" (UID: \"653056a7-146e-44bd-bd1d-41cd75537411\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"
Sep 12 00:44:57.717070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:57.717036    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:57.760594 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:57.760566    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"
Sep 12 00:44:58.031770 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:58.031674    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm"]
Sep 12 00:44:58.034723 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:44:58.034697    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653056a7_146e_44bd_bd1d_41cd75537411.slice/crio-09594c214ba6b5028d41a3ec334db2579f8a0ba704fe4b5eda9d26a9e19e02f9 WatchSource:0}: Error finding container 09594c214ba6b5028d41a3ec334db2579f8a0ba704fe4b5eda9d26a9e19e02f9: Status 404 returned error can't find the container with id 09594c214ba6b5028d41a3ec334db2579f8a0ba704fe4b5eda9d26a9e19e02f9
Sep 12 00:44:58.313183 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:58.313098    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:58.523998 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:58.523966    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm" event={"ID":"653056a7-146e-44bd-bd1d-41cd75537411","Type":"ContainerStarted","Data":"09594c214ba6b5028d41a3ec334db2579f8a0ba704fe4b5eda9d26a9e19e02f9"}
Sep 12 00:44:58.524462 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:58.524190    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:44:58.713032 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:58.712989    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:58.921144 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:58.921109    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:59.102798 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:59.102712    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:59.102798 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:59.102781    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:59.103268 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:59.103248    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b79c5c-195d-436e-8785-90cc7e91d8cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:59.105593 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:59.105568    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b79c5c-195d-436e-8785-90cc7e91d8cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rp657\" (UID: \"19b79c5c-195d-436e-8785-90cc7e91d8cb\") " pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:59.264787 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:59.264746    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rp657"
Sep 12 00:44:59.314167 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:59.314143    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:59.374184 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:59.374138    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:59.625310 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:59.625241    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rp657"]
Sep 12 00:44:59.716137 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:59.716106    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:59.725483 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:44:59.725459    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b79c5c_195d_436e_8785_90cc7e91d8cb.slice/crio-39a9aa8b023c7ec8024206cf75eaf24b6b098add2a1e0f69e6aa099ce7340a4d WatchSource:0}: Error finding container 39a9aa8b023c7ec8024206cf75eaf24b6b098add2a1e0f69e6aa099ce7340a4d: Status 404 returned error can't find the container with id 39a9aa8b023c7ec8024206cf75eaf24b6b098add2a1e0f69e6aa099ce7340a4d
Sep 12 00:44:59.911597 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:44:59.911572    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:00.531730 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:00.531692    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm" event={"ID":"653056a7-146e-44bd-bd1d-41cd75537411","Type":"ContainerStarted","Data":"38bec69f472d45653c8d973f2f69cc8cb0264c680585235b20279e4a759d3f9a"}
Sep 12 00:45:00.533238 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:00.533209    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rp657" event={"ID":"19b79c5c-195d-436e-8785-90cc7e91d8cb","Type":"ContainerStarted","Data":"ab0d733dddea90c5f09062628dbb6307cc6b19c77147d15c4226c21ab63e0967"}
Sep 12 00:45:00.533346 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:00.533257    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rp657" event={"ID":"19b79c5c-195d-436e-8785-90cc7e91d8cb","Type":"ContainerStarted","Data":"39a9aa8b023c7ec8024206cf75eaf24b6b098add2a1e0f69e6aa099ce7340a4d"}
Sep 12 00:45:00.547644 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:00.547603    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-svchm" podStartSLOduration=17.829225816 podStartE2EDuration="19.547585858s" totalImagesPullingTime="1.718360042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:44:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:00.547443563 +0000 UTC m=+105.107895252" watchObservedRunningTime="2026-09-12 00:45:00.547585858 +0000 UTC m=+105.108037522"
Sep 12 00:45:01.538681 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:01.538646    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rp657" event={"ID":"19b79c5c-195d-436e-8785-90cc7e91d8cb","Type":"ContainerStarted","Data":"f61b9ea08169aaaaa7eb1a1fe5688b88a6011432af8c298489db49758c336537"}
Sep 12 00:45:01.635689 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:01.635658    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:01.737377 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:01.737347    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:02.968016 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:02.967974    2567 scope.go:117] "RemoveContainer" containerID="4b28d20ef63a3ef00a4b939fc1c685f45ca430e69f6a69ec55778f69bc25e72b"
Sep 12 00:45:03.546632 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:03.546594    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 00:45:03.546817 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:03.546701    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-gbjdw" event={"ID":"09725ba5-1c2a-4a09-9901-65dcf94d47d4","Type":"ContainerStarted","Data":"84a5a323bd4faa679b2e23fa142ea27ff2c03bacbfcee5f3e32b946b16c0e09f"}
Sep 12 00:45:03.547031 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:03.546996    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:45:03.549152 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:03.549129    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rp657" event={"ID":"19b79c5c-195d-436e-8785-90cc7e91d8cb","Type":"ContainerStarted","Data":"b2b727ad2ed1af616039f697040b028022d5b894b4e148272767b9cde1b9a07e"}
Sep 12 00:45:03.552110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:03.552090    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-gbjdw"
Sep 12 00:45:03.586292 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:03.586260    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:03.594965 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:03.594910    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-gbjdw" podStartSLOduration=22.124742516 podStartE2EDuration="25.594893352s" totalImagesPullingTime="3.470150836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:03.593529847 +0000 UTC m=+108.153981518" watchObservedRunningTime="2026-09-12 00:45:03.594893352 +0000 UTC m=+108.155345019"
Sep 12 00:45:03.629595 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:03.629542    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rp657" podStartSLOduration=18.100191983 podStartE2EDuration="20.62952631s" totalImagesPullingTime="2.529334327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:44:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:03.628529963 +0000 UTC m=+108.188981630" watchObservedRunningTime="2026-09-12 00:45:03.62952631 +0000 UTC m=+108.189977977"
Sep 12 00:45:03.644248 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:03.644219    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:03.709612 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:03.709578    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:04.036620 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:04.036590    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:04.224218 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:04.224179    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:04.286044 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:04.286007    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:04.843864 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:04.843831    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:05.027125 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:05.027092    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:05.623567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:05.623536    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:06.129007 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:06.128972    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:08.197570 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.197539    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-4s92h"]
Sep 12 00:45:08.200655 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.200639    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-4s92h"
Sep 12 00:45:08.204252 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.204230    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5qbfr\""
Sep 12 00:45:08.204404 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.204384    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 12 00:45:08.204487 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.204410    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 12 00:45:08.213638 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.213612    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-4s92h"]
Sep 12 00:45:08.281677 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.281640    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ncn6\" (UniqueName: \"kubernetes.io/projected/e8b9fcfb-2862-479f-b158-7c3c3f4ab57f-kube-api-access-6ncn6\") pod \"downloads-76f99584f4-4s92h\" (UID: \"e8b9fcfb-2862-479f-b158-7c3c3f4ab57f\") " pod="openshift-console/downloads-76f99584f4-4s92h"
Sep 12 00:45:08.289073 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.289034    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:08.304998 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.304975    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58cf8758-nm4j8"]
Sep 12 00:45:08.307890 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.307873    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.311866 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.311842    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 12 00:45:08.312081 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.312069    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rszm7\""
Sep 12 00:45:08.312876 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.312857    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 12 00:45:08.313030 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.313012    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 12 00:45:08.318418 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.318398    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 12 00:45:08.322361 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.322343    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58cf8758-nm4j8"]
Sep 12 00:45:08.382269 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.382240    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e894c61f-cb95-48ed-a6dc-a2d77b165617-image-registry-private-configuration\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.382441 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.382291    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e894c61f-cb95-48ed-a6dc-a2d77b165617-trusted-ca\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.382441 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.382325    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e894c61f-cb95-48ed-a6dc-a2d77b165617-bound-sa-token\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.382441 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.382390    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e894c61f-cb95-48ed-a6dc-a2d77b165617-installation-pull-secrets\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.382441 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.382418    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e894c61f-cb95-48ed-a6dc-a2d77b165617-ca-trust-extracted\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.382591 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.382468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ncn6\" (UniqueName: \"kubernetes.io/projected/e8b9fcfb-2862-479f-b158-7c3c3f4ab57f-kube-api-access-6ncn6\") pod \"downloads-76f99584f4-4s92h\" (UID: \"e8b9fcfb-2862-479f-b158-7c3c3f4ab57f\") " pod="openshift-console/downloads-76f99584f4-4s92h"
Sep 12 00:45:08.382591 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.382488    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e894c61f-cb95-48ed-a6dc-a2d77b165617-registry-certificates\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.382591 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.382504    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx9h5\" (UniqueName: \"kubernetes.io/projected/e894c61f-cb95-48ed-a6dc-a2d77b165617-kube-api-access-cx9h5\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.382591 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.382523    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e894c61f-cb95-48ed-a6dc-a2d77b165617-registry-tls\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.385741 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.385719    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:08.392465 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.392447    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ncn6\" (UniqueName: \"kubernetes.io/projected/e8b9fcfb-2862-479f-b158-7c3c3f4ab57f-kube-api-access-6ncn6\") pod \"downloads-76f99584f4-4s92h\" (UID: \"e8b9fcfb-2862-479f-b158-7c3c3f4ab57f\") " pod="openshift-console/downloads-76f99584f4-4s92h"
Sep 12 00:45:08.483790 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.483693    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e894c61f-cb95-48ed-a6dc-a2d77b165617-registry-certificates\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.483790 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.483736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx9h5\" (UniqueName: \"kubernetes.io/projected/e894c61f-cb95-48ed-a6dc-a2d77b165617-kube-api-access-cx9h5\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.483790 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.483760    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e894c61f-cb95-48ed-a6dc-a2d77b165617-registry-tls\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.483790 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.483784    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e894c61f-cb95-48ed-a6dc-a2d77b165617-image-registry-private-configuration\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.484170 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.483839    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e894c61f-cb95-48ed-a6dc-a2d77b165617-trusted-ca\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.484170 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.483871    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e894c61f-cb95-48ed-a6dc-a2d77b165617-bound-sa-token\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.484170 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.483919    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e894c61f-cb95-48ed-a6dc-a2d77b165617-installation-pull-secrets\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.484170 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.483942    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e894c61f-cb95-48ed-a6dc-a2d77b165617-ca-trust-extracted\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.484426 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.484406    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e894c61f-cb95-48ed-a6dc-a2d77b165617-ca-trust-extracted\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.484707 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.484686    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e894c61f-cb95-48ed-a6dc-a2d77b165617-registry-certificates\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.484764 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.484722    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e894c61f-cb95-48ed-a6dc-a2d77b165617-trusted-ca\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.486402 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.486384    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e894c61f-cb95-48ed-a6dc-a2d77b165617-registry-tls\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.487039 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.487007    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e894c61f-cb95-48ed-a6dc-a2d77b165617-image-registry-private-configuration\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.487131 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.487047    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e894c61f-cb95-48ed-a6dc-a2d77b165617-installation-pull-secrets\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.489100 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.489076    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:08.494977 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.494938    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e894c61f-cb95-48ed-a6dc-a2d77b165617-bound-sa-token\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.498539 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.498516    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx9h5\" (UniqueName: \"kubernetes.io/projected/e894c61f-cb95-48ed-a6dc-a2d77b165617-kube-api-access-cx9h5\") pod \"image-registry-58cf8758-nm4j8\" (UID: \"e894c61f-cb95-48ed-a6dc-a2d77b165617\") " pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.510159 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.510137    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-4s92h"
Sep 12 00:45:08.528835 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.528811    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dsbc2"
Sep 12 00:45:08.590405 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.590328    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:08.616330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.616300    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:08.803863 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.803835    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-4s92h"]
Sep 12 00:45:08.805225 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:45:08.805200    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b9fcfb_2862_479f_b158_7c3c3f4ab57f.slice/crio-b88ab2aced1cc4592329ea9a5dba152ef7de6ff5fff048066b5eb285483c1a72 WatchSource:0}: Error finding container b88ab2aced1cc4592329ea9a5dba152ef7de6ff5fff048066b5eb285483c1a72: Status 404 returned error can't find the container with id b88ab2aced1cc4592329ea9a5dba152ef7de6ff5fff048066b5eb285483c1a72
Sep 12 00:45:08.917746 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.917709    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:08.928863 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:08.928837    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58cf8758-nm4j8"]
Sep 12 00:45:08.931935 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:45:08.931908    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode894c61f_cb95_48ed_a6dc_a2d77b165617.slice/crio-e30020433a04337b58c1a8c9b04f5bcfbc2f78c55089f9789fecfb169e0819f0 WatchSource:0}: Error finding container e30020433a04337b58c1a8c9b04f5bcfbc2f78c55089f9789fecfb169e0819f0: Status 404 returned error can't find the container with id e30020433a04337b58c1a8c9b04f5bcfbc2f78c55089f9789fecfb169e0819f0
Sep 12 00:45:09.008212 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:09.008186    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:09.572364 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:09.572330    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58cf8758-nm4j8" event={"ID":"e894c61f-cb95-48ed-a6dc-a2d77b165617","Type":"ContainerStarted","Data":"d3dbe4eb3c2988aa14bdd403ba38978a3aaa53c6fbbe79a681ed0e23eb979e9a"}
Sep 12 00:45:09.572819 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:09.572378    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58cf8758-nm4j8" event={"ID":"e894c61f-cb95-48ed-a6dc-a2d77b165617","Type":"ContainerStarted","Data":"e30020433a04337b58c1a8c9b04f5bcfbc2f78c55089f9789fecfb169e0819f0"}
Sep 12 00:45:09.572819 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:09.572425    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:09.573853 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:09.573826    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-4s92h" event={"ID":"e8b9fcfb-2862-479f-b158-7c3c3f4ab57f","Type":"ContainerStarted","Data":"b88ab2aced1cc4592329ea9a5dba152ef7de6ff5fff048066b5eb285483c1a72"}
Sep 12 00:45:09.600976 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:09.599467    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58cf8758-nm4j8" podStartSLOduration=1.5994474300000001 podStartE2EDuration="1.59944743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:09.597241535 +0000 UTC m=+114.157693227" watchObservedRunningTime="2026-09-12 00:45:09.59944743 +0000 UTC m=+114.159899099"
Sep 12 00:45:09.865820 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:09.865728    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:10.228112 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:10.228082    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:10.297853 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:10.297821    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:12.218457 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:12.218410    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:12.321781 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:12.321688    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:12.393819 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:12.393779    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:12.905155 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:12.905098    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:15.390070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:15.390028    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:15.460106 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:15.460064    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:16.248241 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:16.248198    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:17.668741 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:17.668709    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:17.725171 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:17.725130    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:18.231496 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:18.231466    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:19.264400 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.264357    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:19.365245 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.365209    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:19.536511 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.536415    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:19.671322 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.671275    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:19.781202 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.781138    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:19.842765 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.842684    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xcjkz"]
Sep 12 00:45:19.847599 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.847570    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:19.850125 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.850072    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 12 00:45:19.850275 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.850233    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 12 00:45:19.851105 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.851068    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 12 00:45:19.851227 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.851167    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 12 00:45:19.851291 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.851236    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-45bhm\""
Sep 12 00:45:19.851360 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.851185    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 12 00:45:19.851725 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.851480    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 12 00:45:19.875622 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.875589    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:19.987074 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.987038    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-tls\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:19.987242 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.987084    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-textfile\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:19.987242 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.987123    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-wtmp\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:19.987242 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.987152    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-metrics-client-ca\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:19.987242 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.987178    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-sys\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:19.987242 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.987231    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-root\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:19.987484 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.987261    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-accelerators-collector-config\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:19.987484 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.987324    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkn9z\" (UniqueName: \"kubernetes.io/projected/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-kube-api-access-tkn9z\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:19.987484 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:19.987355    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.068430 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.068378    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:20.088541 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.088504    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-root\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.088726 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.088552    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-accelerators-collector-config\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.088726 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.088613    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkn9z\" (UniqueName: \"kubernetes.io/projected/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-kube-api-access-tkn9z\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.088726 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.088617    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-root\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.088726 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.088643    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.088726 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.088675    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-tls\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.088726 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.088697    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-textfile\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.089060 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.088729    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-wtmp\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.089060 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.088754    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-metrics-client-ca\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.089060 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.088781    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-sys\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.089060 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.088884    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-sys\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.089060 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.088910    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-wtmp\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.089301 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.089208    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-accelerators-collector-config\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.089563 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.089519    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-metrics-client-ca\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.089563 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.089535    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-textfile\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.091610 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.091591    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-tls\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.091874 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.091854    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.098681 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.098596    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkn9z\" (UniqueName: \"kubernetes.io/projected/36a5e6bc-d6e1-49d4-a665-6d7639fd24b2-kube-api-access-tkn9z\") pod \"node-exporter-xcjkz\" (UID: \"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2\") " pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.167252 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.167223    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xcjkz"
Sep 12 00:45:20.167863 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.167828    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:20.269256 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.269220    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:20.572116 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.572078    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:20.670828 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.670795    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:20.771127 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.771099    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:20.870679 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.870598    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:20.890250 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.890215    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:45:20.894708 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.894682    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:20.898404 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.898374    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 12 00:45:20.898404 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.898384    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 12 00:45:20.899088 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.899067    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 12 00:45:20.899233 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.899150    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 12 00:45:20.899494 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.899462    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 12 00:45:20.899494 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.899470    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 12 00:45:20.899494 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.899470    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 12 00:45:20.899709 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.899475    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-27jp7\""
Sep 12 00:45:20.899881 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.899862    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 12 00:45:20.900419 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.900163    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 12 00:45:20.929873 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.929833    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:45:20.975754 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.975722    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:20.998088 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.998050    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:20.998088 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.998092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:20.998325 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.998118    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:20.998325 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.998136    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a3606e9-aff1-42b4-ae29-1a577c25eb29-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:20.998325 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.998174    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:20.998325 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.998218    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a3606e9-aff1-42b4-ae29-1a577c25eb29-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:20.998325 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.998300    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:20.998515 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.998336    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a3606e9-aff1-42b4-ae29-1a577c25eb29-config-out\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:20.998515 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.998355    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvgj\" (UniqueName: \"kubernetes.io/projected/8a3606e9-aff1-42b4-ae29-1a577c25eb29-kube-api-access-zlvgj\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:20.998515 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.998416    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a3606e9-aff1-42b4-ae29-1a577c25eb29-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:20.998515 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.998435    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-web-config\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:20.998515 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.998453    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:20.998515 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:20.998497    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a3606e9-aff1-42b4-ae29-1a577c25eb29-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.054811 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.054770    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:21.060317 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.060284    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5675b6b7f4-xflw2"]
Sep 12 00:45:21.063854 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.063834    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.066699 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.066682    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mlzth\""
Sep 12 00:45:21.068444 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.068423    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 12 00:45:21.068694 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.068677    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 12 00:45:21.068767 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.068721    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 12 00:45:21.069323 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.069303    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 12 00:45:21.069990 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.069971    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 12 00:45:21.075843 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.075821    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 12 00:45:21.080603 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.080577    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5675b6b7f4-xflw2"]
Sep 12 00:45:21.099601 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.099562    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a3606e9-aff1-42b4-ae29-1a577c25eb29-config-out\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.099601 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.099605    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvgj\" (UniqueName: \"kubernetes.io/projected/8a3606e9-aff1-42b4-ae29-1a577c25eb29-kube-api-access-zlvgj\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.099828 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.099652    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a3606e9-aff1-42b4-ae29-1a577c25eb29-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.099828 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.099676    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-web-config\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.099828 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.099708    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.099828 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.099762    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a3606e9-aff1-42b4-ae29-1a577c25eb29-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.099828 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.099792    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.099828 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.099819    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.100153 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.099848    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.100153 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.099875    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a3606e9-aff1-42b4-ae29-1a577c25eb29-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.100153 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.099915    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.100153 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.099940    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a3606e9-aff1-42b4-ae29-1a577c25eb29-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.100153 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.100001    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.104758 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.104726    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a3606e9-aff1-42b4-ae29-1a577c25eb29-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.104881 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.104801    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.105558 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.105283    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-web-config\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.105558 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.105348    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a3606e9-aff1-42b4-ae29-1a577c25eb29-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.106570 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.106546    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a3606e9-aff1-42b4-ae29-1a577c25eb29-config-out\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.106923 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.106901    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a3606e9-aff1-42b4-ae29-1a577c25eb29-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.108515 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.107992    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.111242 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.109028    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.111242 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.109679    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.111242 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.110316    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.111702 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.111675    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a3606e9-aff1-42b4-ae29-1a577c25eb29-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.113018 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.112991    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.113244 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.113220    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvgj\" (UniqueName: \"kubernetes.io/projected/8a3606e9-aff1-42b4-ae29-1a577c25eb29-kube-api-access-zlvgj\") pod \"alertmanager-main-0\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.203799 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.203755    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-trusted-ca-bundle\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.203999 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.203811    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-service-ca\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.203999 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.203914    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-config\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.204125 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.204009    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-serving-cert\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.204125 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.204040    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-oauth-serving-cert\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.204125 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.204092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv528\" (UniqueName: \"kubernetes.io/projected/5c654414-0339-4096-a9d1-fe3c1c609ec3-kube-api-access-fv528\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.204125 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.204112    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-oauth-config\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.206689 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.206664    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:45:21.304749 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.304700    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-trusted-ca-bundle\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.304749 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.304754    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-service-ca\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.305286 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.304800    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-config\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.305286 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.304837    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-serving-cert\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.305286 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.304868    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-oauth-serving-cert\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.305286 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.305013    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv528\" (UniqueName: \"kubernetes.io/projected/5c654414-0339-4096-a9d1-fe3c1c609ec3-kube-api-access-fv528\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.305286 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.305067    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-oauth-config\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.305619 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.305601    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-service-ca\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.305739 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.305706    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-trusted-ca-bundle\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.306361 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.305889    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-config\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.306361 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.306320    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-oauth-serving-cert\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.307702 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.307679    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-serving-cert\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.308177 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.308155    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-oauth-config\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.313251 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.313232    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv528\" (UniqueName: \"kubernetes.io/projected/5c654414-0339-4096-a9d1-fe3c1c609ec3-kube-api-access-fv528\") pod \"console-5675b6b7f4-xflw2\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") " pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:21.375272 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:21.375232    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:22.784594 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:22.784560    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:22.882890 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:22.882853    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:23.280300 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:23.280269    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:23.452222 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:23.452175    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:24.061150 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.061105    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:24.161823 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.161792    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:24.206202 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.206167    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d58fbf47f-fjh87"]
Sep 12 00:45:24.209782 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.209760    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.213314 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.213287    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3edcv0nhmt87g\""
Sep 12 00:45:24.213444 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.213338    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-t7nx5\""
Sep 12 00:45:24.213444 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.213364    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 12 00:45:24.213444 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.213287    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 12 00:45:24.213444 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.213397    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 12 00:45:24.213444 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.213427    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 12 00:45:24.220858 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.220827    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d58fbf47f-fjh87"]
Sep 12 00:45:24.235078 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.235044    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bc9bb428-0aff-44c1-98a9-4692709adda5-audit-log\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.235271 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.235132    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc9bb428-0aff-44c1-98a9-4692709adda5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.235271 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.235254    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bc9bb428-0aff-44c1-98a9-4692709adda5-secret-metrics-server-client-certs\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.235398 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.235285    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bc9bb428-0aff-44c1-98a9-4692709adda5-client-ca-bundle\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.235398 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.235352    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bc9bb428-0aff-44c1-98a9-4692709adda5-metrics-server-audit-profiles\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.235541 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.235409    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bc9bb428-0aff-44c1-98a9-4692709adda5-secret-metrics-server-tls\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.235541 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.235515    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj6tn\" (UniqueName: \"kubernetes.io/projected/bc9bb428-0aff-44c1-98a9-4692709adda5-kube-api-access-kj6tn\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.336854 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.336738    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bc9bb428-0aff-44c1-98a9-4692709adda5-secret-metrics-server-tls\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.336854 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.336812    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6tn\" (UniqueName: \"kubernetes.io/projected/bc9bb428-0aff-44c1-98a9-4692709adda5-kube-api-access-kj6tn\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.337089 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.336871    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bc9bb428-0aff-44c1-98a9-4692709adda5-audit-log\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.337089 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.336902    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc9bb428-0aff-44c1-98a9-4692709adda5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.337089 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.336940    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bc9bb428-0aff-44c1-98a9-4692709adda5-secret-metrics-server-client-certs\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.337089 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.336991    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bc9bb428-0aff-44c1-98a9-4692709adda5-client-ca-bundle\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.337089 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.337026    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bc9bb428-0aff-44c1-98a9-4692709adda5-metrics-server-audit-profiles\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.337379 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.337351    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bc9bb428-0aff-44c1-98a9-4692709adda5-audit-log\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.337972 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.337928    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bc9bb428-0aff-44c1-98a9-4692709adda5-metrics-server-audit-profiles\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.338060 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.337971    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc9bb428-0aff-44c1-98a9-4692709adda5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.339657 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.339615    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bc9bb428-0aff-44c1-98a9-4692709adda5-secret-metrics-server-tls\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.339780 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.339762    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bc9bb428-0aff-44c1-98a9-4692709adda5-secret-metrics-server-client-certs\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.339857 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.339835    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bc9bb428-0aff-44c1-98a9-4692709adda5-client-ca-bundle\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.345883 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.345839    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj6tn\" (UniqueName: \"kubernetes.io/projected/bc9bb428-0aff-44c1-98a9-4692709adda5-kube-api-access-kj6tn\") pod \"metrics-server-7d58fbf47f-fjh87\" (UID: \"bc9bb428-0aff-44c1-98a9-4692709adda5\") " pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.522058 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.522019    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:24.557974 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.557917    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:24.639914 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.639831    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs\") pod \"network-metrics-daemon-sdhm4\" (UID: \"079a162f-f33c-4911-8385-e6d87ccd2231\") " pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:45:24.642624 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.642566    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079a162f-f33c-4911-8385-e6d87ccd2231-metrics-certs\") pod \"network-metrics-daemon-sdhm4\" (UID: \"079a162f-f33c-4911-8385-e6d87ccd2231\") " pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:45:24.652991 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.652927    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:24.743614 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:45:24.743579    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a5e6bc_d6e1_49d4_a665_6d7639fd24b2.slice/crio-ce3f7975fb8aa42dc755d37aebc40d2b3551b983b5249226faa44eb58bfa2f83 WatchSource:0}: Error finding container ce3f7975fb8aa42dc755d37aebc40d2b3551b983b5249226faa44eb58bfa2f83: Status 404 returned error can't find the container with id ce3f7975fb8aa42dc755d37aebc40d2b3551b983b5249226faa44eb58bfa2f83
Sep 12 00:45:24.802176 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.801745    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:24.890750 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.890671    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-42kqk\""
Sep 12 00:45:24.898464 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:24.898440    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdhm4"
Sep 12 00:45:25.009460 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:25.009418    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5675b6b7f4-xflw2"]
Sep 12 00:45:25.014173 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:45:25.014127    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c654414_0339_4096_a9d1_fe3c1c609ec3.slice/crio-8d81d5c06a7627bac6ec7ae5c5980018f33b49817cf3f7db996dd5069a5d861c WatchSource:0}: Error finding container 8d81d5c06a7627bac6ec7ae5c5980018f33b49817cf3f7db996dd5069a5d861c: Status 404 returned error can't find the container with id 8d81d5c06a7627bac6ec7ae5c5980018f33b49817cf3f7db996dd5069a5d861c
Sep 12 00:45:25.017124 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:25.017053    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d58fbf47f-fjh87"]
Sep 12 00:45:25.019260 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:45:25.019233    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9bb428_0aff_44c1_98a9_4692709adda5.slice/crio-9e906ef2162087fb1306c1d13174fc33fab4b6e88e494ea8a3959bdf114ed034 WatchSource:0}: Error finding container 9e906ef2162087fb1306c1d13174fc33fab4b6e88e494ea8a3959bdf114ed034: Status 404 returned error can't find the container with id 9e906ef2162087fb1306c1d13174fc33fab4b6e88e494ea8a3959bdf114ed034
Sep 12 00:45:25.024532 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:25.024441    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:45:25.024799 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:45:25.024776    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3606e9_aff1_42b4_ae29_1a577c25eb29.slice/crio-522c6d431ddc57e522be0805d26e7e029ac4eb41ffd93e5a81ce4183b49a8e05 WatchSource:0}: Error finding container 522c6d431ddc57e522be0805d26e7e029ac4eb41ffd93e5a81ce4183b49a8e05: Status 404 returned error can't find the container with id 522c6d431ddc57e522be0805d26e7e029ac4eb41ffd93e5a81ce4183b49a8e05
Sep 12 00:45:25.176632 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:25.176472    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sdhm4"]
Sep 12 00:45:25.638905 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:25.638816    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerStarted","Data":"522c6d431ddc57e522be0805d26e7e029ac4eb41ffd93e5a81ce4183b49a8e05"}
Sep 12 00:45:25.640628 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:25.640552    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xcjkz" event={"ID":"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2","Type":"ContainerStarted","Data":"ce3f7975fb8aa42dc755d37aebc40d2b3551b983b5249226faa44eb58bfa2f83"}
Sep 12 00:45:25.642287 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:25.642258    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sdhm4" event={"ID":"079a162f-f33c-4911-8385-e6d87ccd2231","Type":"ContainerStarted","Data":"de85fe46ba376a88e2217e615263bc7de24c7f52c6b3d8ad150680db73d3d817"}
Sep 12 00:45:25.643972 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:25.643925    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87" event={"ID":"bc9bb428-0aff-44c1-98a9-4692709adda5","Type":"ContainerStarted","Data":"9e906ef2162087fb1306c1d13174fc33fab4b6e88e494ea8a3959bdf114ed034"}
Sep 12 00:45:25.667993 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:25.651188    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5675b6b7f4-xflw2" event={"ID":"5c654414-0339-4096-a9d1-fe3c1c609ec3","Type":"ContainerStarted","Data":"8d81d5c06a7627bac6ec7ae5c5980018f33b49817cf3f7db996dd5069a5d861c"}
Sep 12 00:45:25.667993 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:25.655244    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-4s92h" event={"ID":"e8b9fcfb-2862-479f-b158-7c3c3f4ab57f","Type":"ContainerStarted","Data":"e4256864b886ed8c569e7f15e321467cb2c67356fcc86a71e3af4ba6901692a0"}
Sep 12 00:45:25.667993 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:25.657396    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-4s92h"
Sep 12 00:45:25.683692 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:25.683640    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-4s92h"
Sep 12 00:45:25.690610 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:25.690549    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-4s92h" podStartSLOduration=1.657338266 podStartE2EDuration="17.690525572s" totalImagesPullingTime="16.033187306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:25.68717529 +0000 UTC m=+130.247626981" watchObservedRunningTime="2026-09-12 00:45:25.690525572 +0000 UTC m=+130.250977244"
Sep 12 00:45:26.142998 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.142962    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:26.146627 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.146594    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 00:45:26.153006 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.152971    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.158708 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.157897    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 12 00:45:26.158708 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.157981    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 12 00:45:26.158708 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.158165    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jfm55\""
Sep 12 00:45:26.158708 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.158207    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 12 00:45:26.158708 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.158333    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 12 00:45:26.158708 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.158363    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 12 00:45:26.158708 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.158166    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 12 00:45:26.158708 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.158594    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 12 00:45:26.158708 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.158599    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 12 00:45:26.159252 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.158864    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 12 00:45:26.159252 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.157906    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 12 00:45:26.162239 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.161055    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 12 00:45:26.162239 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.161249    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 12 00:45:26.162239 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.161433    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 12 00:45:26.180904 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.162676    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8sdddgerbd5kh\""
Sep 12 00:45:26.180904 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.180809    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 00:45:26.182013 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.181529    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 12 00:45:26.259120 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259081    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.259308 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259131    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg2j8\" (UniqueName: \"kubernetes.io/projected/ccb2247e-8a33-46fd-8006-b12170d71f1e-kube-api-access-bg2j8\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.259308 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259174    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.259308 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259197    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.259308 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259261    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.259308 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259285    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.259308 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259306    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.259610 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259339    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.259610 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259369    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.259610 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259406    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.259610 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259430    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.259610 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259504    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.259610 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259531    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.259610 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259566    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-config\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.259610 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259588    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.260007 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259617    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.260007 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259648    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb2247e-8a33-46fd-8006-b12170d71f1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.260007 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259674    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.260007 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259708    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb2247e-8a33-46fd-8006-b12170d71f1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.260007 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.259733    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.360981 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.360876    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.360981 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.360929    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.360981 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.360987    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-config\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.361852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361013    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.361852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361043    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.361852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361071    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb2247e-8a33-46fd-8006-b12170d71f1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.361852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361099    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.361852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361129    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb2247e-8a33-46fd-8006-b12170d71f1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.361852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361152    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.361852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361643    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.361852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361687    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg2j8\" (UniqueName: \"kubernetes.io/projected/ccb2247e-8a33-46fd-8006-b12170d71f1e-kube-api-access-bg2j8\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.361852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361760    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.361852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361788    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.361852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361840    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.362337 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361864    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.362337 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361894    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.362337 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361936    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.362337 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.361988    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.362337 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.362033    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.362337 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.362061    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.366030 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.364731    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.366030 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.365379    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.366199 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.366027    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.367986 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.366611    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.367986 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.366624    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.367986 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.367828    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.367986 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.367932    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.371463 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.371441    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.374456 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.374314    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.374771 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.374704    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.381996 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.381976    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.383807 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.383461    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.384525 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.384482    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg2j8\" (UniqueName: \"kubernetes.io/projected/ccb2247e-8a33-46fd-8006-b12170d71f1e-kube-api-access-bg2j8\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.393479 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.393449    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.397823 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.397794    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb2247e-8a33-46fd-8006-b12170d71f1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.398389 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.398370    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb2247e-8a33-46fd-8006-b12170d71f1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.398487 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.398431    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-config\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.398792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.398768    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.399022 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.399003    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.401310 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.401288    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.494018 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.493628    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:26.664268 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.664230    2567 generic.go:356] "Generic (PLEG): container finished" podID="36a5e6bc-d6e1-49d4-a665-6d7639fd24b2" containerID="35e26de1510babe1d39f49d4cb2350a36ba954a937d5a0712d44d7564ebdd109" exitCode=0
Sep 12 00:45:26.665077 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.665048    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xcjkz" event={"ID":"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2","Type":"ContainerDied","Data":"35e26de1510babe1d39f49d4cb2350a36ba954a937d5a0712d44d7564ebdd109"}
Sep 12 00:45:26.832066 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:26.831879    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 00:45:26.841854 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:45:26.841820    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb2247e_8a33_46fd_8006_b12170d71f1e.slice/crio-de43306b473e3b1784a6749b4722627c0059f0e68bb53a364ce3e711feb825ab WatchSource:0}: Error finding container de43306b473e3b1784a6749b4722627c0059f0e68bb53a364ce3e711feb825ab: Status 404 returned error can't find the container with id de43306b473e3b1784a6749b4722627c0059f0e68bb53a364ce3e711feb825ab
Sep 12 00:45:27.500060 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:27.499931    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5675b6b7f4-xflw2"]
Sep 12 00:45:27.674153 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:27.674075    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xcjkz" event={"ID":"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2","Type":"ContainerStarted","Data":"c4f88eb20d4a39b713ebe7b236ac765b8a1c99e2dca4151dc3ebe020af6d2119"}
Sep 12 00:45:27.678083 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:27.677988    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerStarted","Data":"de43306b473e3b1784a6749b4722627c0059f0e68bb53a364ce3e711feb825ab"}
Sep 12 00:45:28.358519 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:28.358478    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:28.417723 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:28.417690    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:29.052780 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:29.052743    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:30.582971 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:30.582921    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58cf8758-nm4j8"
Sep 12 00:45:30.724552 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:30.724521    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:30.825564 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:30.825484    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:31.647170 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.647129    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:31.699996 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.699963    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5675b6b7f4-xflw2" event={"ID":"5c654414-0339-4096-a9d1-fe3c1c609ec3","Type":"ContainerStarted","Data":"de5d9b549ef9ebe1f5a182c881baecac934faf116843da7f4778ae2f80fcafc2"}
Sep 12 00:45:31.702245 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.702213    2567 generic.go:356] "Generic (PLEG): container finished" podID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerID="e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0" exitCode=0
Sep 12 00:45:31.702397 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.702324    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerDied","Data":"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"}
Sep 12 00:45:31.706608 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.706579    2567 generic.go:356] "Generic (PLEG): container finished" podID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerID="655ed296c58473237243162066accf52e25ac1622c2516522b8a921bbaebc38b" exitCode=0
Sep 12 00:45:31.706724 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.706661    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerDied","Data":"655ed296c58473237243162066accf52e25ac1622c2516522b8a921bbaebc38b"}
Sep 12 00:45:31.710012 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.709984    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xcjkz" event={"ID":"36a5e6bc-d6e1-49d4-a665-6d7639fd24b2","Type":"ContainerStarted","Data":"ab9d98387681fb81df0b3a0eb122301d1783c8dcdfbf84fc7abb6af96d32bcc1"}
Sep 12 00:45:31.713524 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.713491    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sdhm4" event={"ID":"079a162f-f33c-4911-8385-e6d87ccd2231","Type":"ContainerStarted","Data":"9d7ad6e98d599b8460448aad9b3700fcdeaf0a8ba1bda3823db8dd957b97ddc2"}
Sep 12 00:45:31.713692 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.713672    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sdhm4" event={"ID":"079a162f-f33c-4911-8385-e6d87ccd2231","Type":"ContainerStarted","Data":"2853b6cb9377c85cf206a3b543e1abef4d1136248e6d81cfea1a33bc5a029c8f"}
Sep 12 00:45:31.715814 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.715788    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87" event={"ID":"bc9bb428-0aff-44c1-98a9-4692709adda5","Type":"ContainerStarted","Data":"80a4ce40145b8050034c925eb4b93d457cc8455de2535186164800a487b4dc4b"}
Sep 12 00:45:31.721390 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.721344    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5675b6b7f4-xflw2" podStartSLOduration=5.052935904 podStartE2EDuration="10.721326846s" totalImagesPullingTime="5.668390942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:31.719527294 +0000 UTC m=+136.279978984" watchObservedRunningTime="2026-09-12 00:45:31.721326846 +0000 UTC m=+136.281778518"
Sep 12 00:45:31.727402 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.727379    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:31.743313 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.743260    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sdhm4" podStartSLOduration=130.245617498 podStartE2EDuration="2m15.743241343s" totalImagesPullingTime="5.497623845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:31.742645881 +0000 UTC m=+136.303097551" watchObservedRunningTime="2026-09-12 00:45:31.743241343 +0000 UTC m=+136.303693017"
Sep 12 00:45:31.816398 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.816337    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87" podStartSLOduration=2.1520232 podStartE2EDuration="7.816316083s" totalImagesPullingTime="5.664292883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:31.815307205 +0000 UTC m=+136.375758875" watchObservedRunningTime="2026-09-12 00:45:31.816316083 +0000 UTC m=+136.376767754"
Sep 12 00:45:31.869128 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.869071    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xcjkz" podStartSLOduration=11.623090289 podStartE2EDuration="12.869052487s" totalImagesPullingTime="1.132209486s" totalInitContainerRuntime="113.752712ms" isStatefulPod=true podCreationTimestamp="2026-09-12 00:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:31.866329105 +0000 UTC m=+136.426780791" watchObservedRunningTime="2026-09-12 00:45:31.869052487 +0000 UTC m=+136.429504155"
Sep 12 00:45:31.950896 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:31.950857    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:32.045686 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:32.045634    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:34.741800 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:34.741765    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerStarted","Data":"10e7284f9bb61867e99d23fc0727b2de94d84db6146af11b3ff857da8a57eeb1"}
Sep 12 00:45:35.751071 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:35.750903    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerStarted","Data":"b770ac3dd989e5d53c0c84ec9e36ba801afacecce27c15eb247d01bf2bda419a"}
Sep 12 00:45:35.751071 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:35.751023    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerStarted","Data":"cebee570b2eedcd77744cadbf65765550c057e427bc32b01b94a465bd704367f"}
Sep 12 00:45:35.751506 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:35.751074    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerStarted","Data":"8a8d8ce04eb8d044012820873915e510e3daef8fc737d4b64a84af7aa71d3a83"}
Sep 12 00:45:35.762182 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:35.762082    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerStarted","Data":"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"}
Sep 12 00:45:36.769322 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:36.769286    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerStarted","Data":"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"}
Sep 12 00:45:36.772844 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:36.772813    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerStarted","Data":"5e9b7ca968622fd4569bf8b17e29f5029baddf0a02c2c22ae678e824af73b062"}
Sep 12 00:45:37.780587 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:37.780509    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerStarted","Data":"fb63cbe438ad6806ac6cbbb3e07921976954ce8b73fc9d0c130b71f73ed7ce8b"}
Sep 12 00:45:37.811324 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:37.811272    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=8.056343296 podStartE2EDuration="17.811256592s" totalImagesPullingTime="9.72519378s" totalInitContainerRuntime="29.719516ms" isStatefulPod=false podCreationTimestamp="2026-09-12 00:45:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:37.808764474 +0000 UTC m=+142.369216188" watchObservedRunningTime="2026-09-12 00:45:37.811256592 +0000 UTC m=+142.371708305"
Sep 12 00:45:38.787557 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:38.787521    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerStarted","Data":"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"}
Sep 12 00:45:38.788051 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:38.787614    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerStarted","Data":"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"}
Sep 12 00:45:38.788051 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:38.787653    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerStarted","Data":"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"}
Sep 12 00:45:38.788051 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:38.787677    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerStarted","Data":"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"}
Sep 12 00:45:38.823653 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:38.823584    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.035933432 podStartE2EDuration="12.823562596s" totalImagesPullingTime="9.778638839s" totalInitContainerRuntime="8.990325ms" isStatefulPod=false podCreationTimestamp="2026-09-12 00:45:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:38.82122802 +0000 UTC m=+143.381679711" watchObservedRunningTime="2026-09-12 00:45:38.823562596 +0000 UTC m=+143.384014264"
Sep 12 00:45:41.375908 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:41.375868    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:41.494577 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:41.494532    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:45:42.129581 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:42.129550    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:42.233793 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:42.233748    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:42.497188 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:42.497144    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:42.598192 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:42.598164    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:42.659146 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:42.659100    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:44.522355 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:44.522317    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:44.523082 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:44.522393    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:45:46.634345 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:46.634307    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:46.694941 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:46.694902    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:47.585266 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:47.585236    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:47.685671 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:47.685633    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:47.745284 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:47.745251    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:56.728830 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:56.728773    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5675b6b7f4-xflw2" podUID="5c654414-0339-4096-a9d1-fe3c1c609ec3" containerName="console" containerID="cri-o://de5d9b549ef9ebe1f5a182c881baecac934faf116843da7f4778ae2f80fcafc2" gracePeriod=15
Sep 12 00:45:56.872196 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:56.872144    2567 generic.go:356] "Generic (PLEG): container finished" podID="5c654414-0339-4096-a9d1-fe3c1c609ec3" containerID="de5d9b549ef9ebe1f5a182c881baecac934faf116843da7f4778ae2f80fcafc2" exitCode=0
Sep 12 00:45:56.872391 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:56.872229    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5675b6b7f4-xflw2" event={"ID":"5c654414-0339-4096-a9d1-fe3c1c609ec3","Type":"ContainerDied","Data":"de5d9b549ef9ebe1f5a182c881baecac934faf116843da7f4778ae2f80fcafc2"}
Sep 12 00:45:56.973687 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:56.973664    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:57.067246 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.067160    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-serving-cert\") pod \"5c654414-0339-4096-a9d1-fe3c1c609ec3\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") "
Sep 12 00:45:57.067246 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.067210    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-service-ca\") pod \"5c654414-0339-4096-a9d1-fe3c1c609ec3\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") "
Sep 12 00:45:57.067246 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.067229    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv528\" (UniqueName: \"kubernetes.io/projected/5c654414-0339-4096-a9d1-fe3c1c609ec3-kube-api-access-fv528\") pod \"5c654414-0339-4096-a9d1-fe3c1c609ec3\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") "
Sep 12 00:45:57.067526 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.067265    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-config\") pod \"5c654414-0339-4096-a9d1-fe3c1c609ec3\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") "
Sep 12 00:45:57.067526 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.067293    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-trusted-ca-bundle\") pod \"5c654414-0339-4096-a9d1-fe3c1c609ec3\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") "
Sep 12 00:45:57.067526 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.067327    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-oauth-config\") pod \"5c654414-0339-4096-a9d1-fe3c1c609ec3\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") "
Sep 12 00:45:57.067526 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.067384    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-oauth-serving-cert\") pod \"5c654414-0339-4096-a9d1-fe3c1c609ec3\" (UID: \"5c654414-0339-4096-a9d1-fe3c1c609ec3\") "
Sep 12 00:45:57.067728 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.067644    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-service-ca" (OuterVolumeSpecName: "service-ca") pod "5c654414-0339-4096-a9d1-fe3c1c609ec3" (UID: "5c654414-0339-4096-a9d1-fe3c1c609ec3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:45:57.067780 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.067756    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-config" (OuterVolumeSpecName: "console-config") pod "5c654414-0339-4096-a9d1-fe3c1c609ec3" (UID: "5c654414-0339-4096-a9d1-fe3c1c609ec3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:45:57.068226 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.068203    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5c654414-0339-4096-a9d1-fe3c1c609ec3" (UID: "5c654414-0339-4096-a9d1-fe3c1c609ec3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:45:57.068354 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.068225    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5c654414-0339-4096-a9d1-fe3c1c609ec3" (UID: "5c654414-0339-4096-a9d1-fe3c1c609ec3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:45:57.069802 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.069773    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5c654414-0339-4096-a9d1-fe3c1c609ec3" (UID: "5c654414-0339-4096-a9d1-fe3c1c609ec3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:45:57.069906 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.069813    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c654414-0339-4096-a9d1-fe3c1c609ec3-kube-api-access-fv528" (OuterVolumeSpecName: "kube-api-access-fv528") pod "5c654414-0339-4096-a9d1-fe3c1c609ec3" (UID: "5c654414-0339-4096-a9d1-fe3c1c609ec3"). InnerVolumeSpecName "kube-api-access-fv528". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:45:57.070020 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.069987    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5c654414-0339-4096-a9d1-fe3c1c609ec3" (UID: "5c654414-0339-4096-a9d1-fe3c1c609ec3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:45:57.168372 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.168321    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-oauth-config\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:57.168372 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.168366    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-oauth-serving-cert\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:57.168372 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.168376    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-serving-cert\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:57.168372 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.168385    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:57.168810 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.168395    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv528\" (UniqueName: \"kubernetes.io/projected/5c654414-0339-4096-a9d1-fe3c1c609ec3-kube-api-access-fv528\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:57.168810 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.168404    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-console-config\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:57.168810 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.168412    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c654414-0339-4096-a9d1-fe3c1c609ec3-trusted-ca-bundle\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:57.877639 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.877608    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5675b6b7f4-xflw2"
Sep 12 00:45:57.877639 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.877629    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5675b6b7f4-xflw2" event={"ID":"5c654414-0339-4096-a9d1-fe3c1c609ec3","Type":"ContainerDied","Data":"8d81d5c06a7627bac6ec7ae5c5980018f33b49817cf3f7db996dd5069a5d861c"}
Sep 12 00:45:57.878174 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.877692    2567 scope.go:117] "RemoveContainer" containerID="de5d9b549ef9ebe1f5a182c881baecac934faf116843da7f4778ae2f80fcafc2"
Sep 12 00:45:57.898014 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.897977    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5675b6b7f4-xflw2"]
Sep 12 00:45:57.902004 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.901982    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5675b6b7f4-xflw2"]
Sep 12 00:45:57.972523 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:57.972492    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c654414-0339-4096-a9d1-fe3c1c609ec3" path="/var/lib/kubelet/pods/5c654414-0339-4096-a9d1-fe3c1c609ec3/volumes"
Sep 12 00:45:58.883258 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:58.883228    2567 generic.go:356] "Generic (PLEG): container finished" podID="48b0b2b4-0df2-410f-89d0-705dd447dc8a" containerID="eec5d059bebdbf6b4ec92f421d8382abff95863f808be5810e488aa89501e186" exitCode=0
Sep 12 00:45:58.883731 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:58.883309    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz" event={"ID":"48b0b2b4-0df2-410f-89d0-705dd447dc8a","Type":"ContainerDied","Data":"eec5d059bebdbf6b4ec92f421d8382abff95863f808be5810e488aa89501e186"}
Sep 12 00:45:58.883731 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:58.883664    2567 scope.go:117] "RemoveContainer" containerID="eec5d059bebdbf6b4ec92f421d8382abff95863f808be5810e488aa89501e186"
Sep 12 00:45:59.329629 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:59.329591    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:59.390742 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:59.390713    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:59.890036 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:45:59.889998    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g22zz" event={"ID":"48b0b2b4-0df2-410f-89d0-705dd447dc8a","Type":"ContainerStarted","Data":"d19d0e6214ec357b97752ec8101bdc7e2692b8d463593c61758fce4b281704ab"}
Sep 12 00:46:01.635853 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:01.635818    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:01.695396 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:01.695360    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:04.527272 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:04.527243    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:46:04.531118 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:04.531094    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d58fbf47f-fjh87"
Sep 12 00:46:07.924898 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:07.924863    2567 generic.go:356] "Generic (PLEG): container finished" podID="aec6dfd1-8c25-4904-9234-23f02c32ea46" containerID="dac3444d9620da266adc426ae75d735a7ab342de7520324466cb32d978a43e1b" exitCode=0
Sep 12 00:46:07.925349 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:07.924935    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-26rk9" event={"ID":"aec6dfd1-8c25-4904-9234-23f02c32ea46","Type":"ContainerDied","Data":"dac3444d9620da266adc426ae75d735a7ab342de7520324466cb32d978a43e1b"}
Sep 12 00:46:07.925401 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:07.925380    2567 scope.go:117] "RemoveContainer" containerID="dac3444d9620da266adc426ae75d735a7ab342de7520324466cb32d978a43e1b"
Sep 12 00:46:07.926533 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:07.926518    2567 generic.go:356] "Generic (PLEG): container finished" podID="a58e1e17-be8b-47d5-bdee-cd38e5554c44" containerID="f648122c50dbee3e03c4a75a587852ae5ee7074eb19a4a12286762fd1299c6ec" exitCode=0
Sep 12 00:46:07.926588 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:07.926577    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt" event={"ID":"a58e1e17-be8b-47d5-bdee-cd38e5554c44","Type":"ContainerDied","Data":"f648122c50dbee3e03c4a75a587852ae5ee7074eb19a4a12286762fd1299c6ec"}
Sep 12 00:46:07.926823 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:07.926811    2567 scope.go:117] "RemoveContainer" containerID="f648122c50dbee3e03c4a75a587852ae5ee7074eb19a4a12286762fd1299c6ec"
Sep 12 00:46:08.311155 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:08.311057    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:08.412084 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:08.412019    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:08.932653 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:08.932614    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-26rk9" event={"ID":"aec6dfd1-8c25-4904-9234-23f02c32ea46","Type":"ContainerStarted","Data":"4c4bbfc2f29b87e59af6ea38523f0f06cd2abfac21b42ab9b48f22f6a69f5c87"}
Sep 12 00:46:08.934799 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:08.934771    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5pnzt" event={"ID":"a58e1e17-be8b-47d5-bdee-cd38e5554c44","Type":"ContainerStarted","Data":"5fce10e90e8e9e775d0d1b62752f97663b74b2fc057b2cb37a0675c8eb6c601e"}
Sep 12 00:46:25.195338 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:25.195297    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:26.494255 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:26.494216    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:26.513866 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:26.513827    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:27.022163 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:27.022137    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:36.006784 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:36.006745    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:36.064007 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:36.063932    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:38.782447 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:38.782408    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:40.232173 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:40.232137    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:46:40.232601 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:40.232409    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="alertmanager" containerID="cri-o://10e7284f9bb61867e99d23fc0727b2de94d84db6146af11b3ff857da8a57eeb1" gracePeriod=120
Sep 12 00:46:40.232601 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:40.232502    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="kube-rbac-proxy-metric" containerID="cri-o://5e9b7ca968622fd4569bf8b17e29f5029baddf0a02c2c22ae678e824af73b062" gracePeriod=120
Sep 12 00:46:40.232601 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:40.232528    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="config-reloader" containerID="cri-o://8a8d8ce04eb8d044012820873915e510e3daef8fc737d4b64a84af7aa71d3a83" gracePeriod=120
Sep 12 00:46:40.232819 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:40.232579    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="prom-label-proxy" containerID="cri-o://fb63cbe438ad6806ac6cbbb3e07921976954ce8b73fc9d0c130b71f73ed7ce8b" gracePeriod=120
Sep 12 00:46:40.232819 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:40.232510    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="kube-rbac-proxy-web" containerID="cri-o://cebee570b2eedcd77744cadbf65765550c057e427bc32b01b94a465bd704367f" gracePeriod=120
Sep 12 00:46:40.232819 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:40.232589    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="kube-rbac-proxy" containerID="cri-o://b770ac3dd989e5d53c0c84ec9e36ba801afacecce27c15eb247d01bf2bda419a" gracePeriod=120
Sep 12 00:46:41.068346 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.068314    2567 generic.go:356] "Generic (PLEG): container finished" podID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerID="fb63cbe438ad6806ac6cbbb3e07921976954ce8b73fc9d0c130b71f73ed7ce8b" exitCode=0
Sep 12 00:46:41.068346 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.068340    2567 generic.go:356] "Generic (PLEG): container finished" podID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerID="5e9b7ca968622fd4569bf8b17e29f5029baddf0a02c2c22ae678e824af73b062" exitCode=0
Sep 12 00:46:41.068346 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.068348    2567 generic.go:356] "Generic (PLEG): container finished" podID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerID="b770ac3dd989e5d53c0c84ec9e36ba801afacecce27c15eb247d01bf2bda419a" exitCode=0
Sep 12 00:46:41.068346 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.068359    2567 generic.go:356] "Generic (PLEG): container finished" podID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerID="8a8d8ce04eb8d044012820873915e510e3daef8fc737d4b64a84af7aa71d3a83" exitCode=0
Sep 12 00:46:41.068346 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.068366    2567 generic.go:356] "Generic (PLEG): container finished" podID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerID="10e7284f9bb61867e99d23fc0727b2de94d84db6146af11b3ff857da8a57eeb1" exitCode=0
Sep 12 00:46:41.068659 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.068387    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerDied","Data":"fb63cbe438ad6806ac6cbbb3e07921976954ce8b73fc9d0c130b71f73ed7ce8b"}
Sep 12 00:46:41.068659 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.068451    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerDied","Data":"5e9b7ca968622fd4569bf8b17e29f5029baddf0a02c2c22ae678e824af73b062"}
Sep 12 00:46:41.068659 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.068478    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerDied","Data":"b770ac3dd989e5d53c0c84ec9e36ba801afacecce27c15eb247d01bf2bda419a"}
Sep 12 00:46:41.068659 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.068503    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerDied","Data":"8a8d8ce04eb8d044012820873915e510e3daef8fc737d4b64a84af7aa71d3a83"}
Sep 12 00:46:41.068659 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.068526    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerDied","Data":"10e7284f9bb61867e99d23fc0727b2de94d84db6146af11b3ff857da8a57eeb1"}
Sep 12 00:46:41.482181 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.482157    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:46:41.652843 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.652756    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-web-config\") pod \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") "
Sep 12 00:46:41.652843 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.652798    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlvgj\" (UniqueName: \"kubernetes.io/projected/8a3606e9-aff1-42b4-ae29-1a577c25eb29-kube-api-access-zlvgj\") pod \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") "
Sep 12 00:46:41.652843 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.652817    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-cluster-tls-config\") pod \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") "
Sep 12 00:46:41.652843 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.652843    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a3606e9-aff1-42b4-ae29-1a577c25eb29-config-out\") pod \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") "
Sep 12 00:46:41.653135 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.653007    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a3606e9-aff1-42b4-ae29-1a577c25eb29-tls-assets\") pod \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") "
Sep 12 00:46:41.653135 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.653078    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") "
Sep 12 00:46:41.653135 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.653106    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-config-volume\") pod \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") "
Sep 12 00:46:41.653135 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.653129    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a3606e9-aff1-42b4-ae29-1a577c25eb29-metrics-client-ca\") pod \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") "
Sep 12 00:46:41.653342 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.653164    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy\") pod \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") "
Sep 12 00:46:41.653342 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.653195    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-main-tls\") pod \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") "
Sep 12 00:46:41.653342 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.653248    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy-web\") pod \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") "
Sep 12 00:46:41.653342 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.653287    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a3606e9-aff1-42b4-ae29-1a577c25eb29-alertmanager-trusted-ca-bundle\") pod \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") "
Sep 12 00:46:41.653342 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.653335    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a3606e9-aff1-42b4-ae29-1a577c25eb29-alertmanager-main-db\") pod \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\" (UID: \"8a3606e9-aff1-42b4-ae29-1a577c25eb29\") "
Sep 12 00:46:41.654240 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.654111    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3606e9-aff1-42b4-ae29-1a577c25eb29-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8a3606e9-aff1-42b4-ae29-1a577c25eb29" (UID: "8a3606e9-aff1-42b4-ae29-1a577c25eb29"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:41.654688 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.654433    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3606e9-aff1-42b4-ae29-1a577c25eb29-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8a3606e9-aff1-42b4-ae29-1a577c25eb29" (UID: "8a3606e9-aff1-42b4-ae29-1a577c25eb29"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:46:41.655516 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.655489    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3606e9-aff1-42b4-ae29-1a577c25eb29-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8a3606e9-aff1-42b4-ae29-1a577c25eb29" (UID: "8a3606e9-aff1-42b4-ae29-1a577c25eb29"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:46:41.656070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.656030    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3606e9-aff1-42b4-ae29-1a577c25eb29-kube-api-access-zlvgj" (OuterVolumeSpecName: "kube-api-access-zlvgj") pod "8a3606e9-aff1-42b4-ae29-1a577c25eb29" (UID: "8a3606e9-aff1-42b4-ae29-1a577c25eb29"). InnerVolumeSpecName "kube-api-access-zlvgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:46:41.656584 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.656518    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-config-volume" (OuterVolumeSpecName: "config-volume") pod "8a3606e9-aff1-42b4-ae29-1a577c25eb29" (UID: "8a3606e9-aff1-42b4-ae29-1a577c25eb29"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:41.656584 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.656550    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3606e9-aff1-42b4-ae29-1a577c25eb29-config-out" (OuterVolumeSpecName: "config-out") pod "8a3606e9-aff1-42b4-ae29-1a577c25eb29" (UID: "8a3606e9-aff1-42b4-ae29-1a577c25eb29"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:41.656584 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.656555    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3606e9-aff1-42b4-ae29-1a577c25eb29-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8a3606e9-aff1-42b4-ae29-1a577c25eb29" (UID: "8a3606e9-aff1-42b4-ae29-1a577c25eb29"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:46:41.656942 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.656917    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8a3606e9-aff1-42b4-ae29-1a577c25eb29" (UID: "8a3606e9-aff1-42b4-ae29-1a577c25eb29"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:41.657461 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.657434    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8a3606e9-aff1-42b4-ae29-1a577c25eb29" (UID: "8a3606e9-aff1-42b4-ae29-1a577c25eb29"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:41.657792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.657757    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8a3606e9-aff1-42b4-ae29-1a577c25eb29" (UID: "8a3606e9-aff1-42b4-ae29-1a577c25eb29"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:41.657995 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.657968    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8a3606e9-aff1-42b4-ae29-1a577c25eb29" (UID: "8a3606e9-aff1-42b4-ae29-1a577c25eb29"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:41.660597 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.660571    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8a3606e9-aff1-42b4-ae29-1a577c25eb29" (UID: "8a3606e9-aff1-42b4-ae29-1a577c25eb29"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:41.667609 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.667586    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-web-config" (OuterVolumeSpecName: "web-config") pod "8a3606e9-aff1-42b4-ae29-1a577c25eb29" (UID: "8a3606e9-aff1-42b4-ae29-1a577c25eb29"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:41.754383 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.754350    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:41.754383 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.754379    2567 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-config-volume\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:41.754383 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.754388    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a3606e9-aff1-42b4-ae29-1a577c25eb29-metrics-client-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:41.754607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.754397    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:41.754607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.754409    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-main-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:41.754607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.754418    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:41.754607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.754426    2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a3606e9-aff1-42b4-ae29-1a577c25eb29-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:41.754607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.754434    2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a3606e9-aff1-42b4-ae29-1a577c25eb29-alertmanager-main-db\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:41.754607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.754463    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-web-config\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:41.754607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.754472    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlvgj\" (UniqueName: \"kubernetes.io/projected/8a3606e9-aff1-42b4-ae29-1a577c25eb29-kube-api-access-zlvgj\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:41.754607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.754479    2567 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a3606e9-aff1-42b4-ae29-1a577c25eb29-cluster-tls-config\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:41.754607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.754488    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a3606e9-aff1-42b4-ae29-1a577c25eb29-config-out\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:41.754607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:41.754497    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a3606e9-aff1-42b4-ae29-1a577c25eb29-tls-assets\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:42.076212 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.076175    2567 generic.go:356] "Generic (PLEG): container finished" podID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerID="cebee570b2eedcd77744cadbf65765550c057e427bc32b01b94a465bd704367f" exitCode=0
Sep 12 00:46:42.076377 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.076243    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerDied","Data":"cebee570b2eedcd77744cadbf65765550c057e427bc32b01b94a465bd704367f"}
Sep 12 00:46:42.076377 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.076259    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:46:42.076377 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.076307    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a3606e9-aff1-42b4-ae29-1a577c25eb29","Type":"ContainerDied","Data":"522c6d431ddc57e522be0805d26e7e029ac4eb41ffd93e5a81ce4183b49a8e05"}
Sep 12 00:46:42.076377 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.076340    2567 scope.go:117] "RemoveContainer" containerID="fb63cbe438ad6806ac6cbbb3e07921976954ce8b73fc9d0c130b71f73ed7ce8b"
Sep 12 00:46:42.084134 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.084112    2567 scope.go:117] "RemoveContainer" containerID="5e9b7ca968622fd4569bf8b17e29f5029baddf0a02c2c22ae678e824af73b062"
Sep 12 00:46:42.091182 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.091164    2567 scope.go:117] "RemoveContainer" containerID="b770ac3dd989e5d53c0c84ec9e36ba801afacecce27c15eb247d01bf2bda419a"
Sep 12 00:46:42.097564 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.097542    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:46:42.098719 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.098704    2567 scope.go:117] "RemoveContainer" containerID="cebee570b2eedcd77744cadbf65765550c057e427bc32b01b94a465bd704367f"
Sep 12 00:46:42.101722 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.101702    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:46:42.106235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.106218    2567 scope.go:117] "RemoveContainer" containerID="8a8d8ce04eb8d044012820873915e510e3daef8fc737d4b64a84af7aa71d3a83"
Sep 12 00:46:42.113155 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.113137    2567 scope.go:117] "RemoveContainer" containerID="10e7284f9bb61867e99d23fc0727b2de94d84db6146af11b3ff857da8a57eeb1"
Sep 12 00:46:42.120581 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.120566    2567 scope.go:117] "RemoveContainer" containerID="655ed296c58473237243162066accf52e25ac1622c2516522b8a921bbaebc38b"
Sep 12 00:46:42.127769 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.127753    2567 scope.go:117] "RemoveContainer" containerID="fb63cbe438ad6806ac6cbbb3e07921976954ce8b73fc9d0c130b71f73ed7ce8b"
Sep 12 00:46:42.128134 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:46:42.128113    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb63cbe438ad6806ac6cbbb3e07921976954ce8b73fc9d0c130b71f73ed7ce8b\": container with ID starting with fb63cbe438ad6806ac6cbbb3e07921976954ce8b73fc9d0c130b71f73ed7ce8b not found: ID does not exist" containerID="fb63cbe438ad6806ac6cbbb3e07921976954ce8b73fc9d0c130b71f73ed7ce8b"
Sep 12 00:46:42.128188 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.128146    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb63cbe438ad6806ac6cbbb3e07921976954ce8b73fc9d0c130b71f73ed7ce8b"} err="failed to get container status \"fb63cbe438ad6806ac6cbbb3e07921976954ce8b73fc9d0c130b71f73ed7ce8b\": rpc error: code = NotFound desc = could not find container \"fb63cbe438ad6806ac6cbbb3e07921976954ce8b73fc9d0c130b71f73ed7ce8b\": container with ID starting with fb63cbe438ad6806ac6cbbb3e07921976954ce8b73fc9d0c130b71f73ed7ce8b not found: ID does not exist"
Sep 12 00:46:42.128188 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.128180    2567 scope.go:117] "RemoveContainer" containerID="5e9b7ca968622fd4569bf8b17e29f5029baddf0a02c2c22ae678e824af73b062"
Sep 12 00:46:42.128481 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:46:42.128464    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9b7ca968622fd4569bf8b17e29f5029baddf0a02c2c22ae678e824af73b062\": container with ID starting with 5e9b7ca968622fd4569bf8b17e29f5029baddf0a02c2c22ae678e824af73b062 not found: ID does not exist" containerID="5e9b7ca968622fd4569bf8b17e29f5029baddf0a02c2c22ae678e824af73b062"
Sep 12 00:46:42.128524 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.128487    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9b7ca968622fd4569bf8b17e29f5029baddf0a02c2c22ae678e824af73b062"} err="failed to get container status \"5e9b7ca968622fd4569bf8b17e29f5029baddf0a02c2c22ae678e824af73b062\": rpc error: code = NotFound desc = could not find container \"5e9b7ca968622fd4569bf8b17e29f5029baddf0a02c2c22ae678e824af73b062\": container with ID starting with 5e9b7ca968622fd4569bf8b17e29f5029baddf0a02c2c22ae678e824af73b062 not found: ID does not exist"
Sep 12 00:46:42.128524 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.128504    2567 scope.go:117] "RemoveContainer" containerID="b770ac3dd989e5d53c0c84ec9e36ba801afacecce27c15eb247d01bf2bda419a"
Sep 12 00:46:42.128726 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:46:42.128711    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b770ac3dd989e5d53c0c84ec9e36ba801afacecce27c15eb247d01bf2bda419a\": container with ID starting with b770ac3dd989e5d53c0c84ec9e36ba801afacecce27c15eb247d01bf2bda419a not found: ID does not exist" containerID="b770ac3dd989e5d53c0c84ec9e36ba801afacecce27c15eb247d01bf2bda419a"
Sep 12 00:46:42.128768 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.128732    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b770ac3dd989e5d53c0c84ec9e36ba801afacecce27c15eb247d01bf2bda419a"} err="failed to get container status \"b770ac3dd989e5d53c0c84ec9e36ba801afacecce27c15eb247d01bf2bda419a\": rpc error: code = NotFound desc = could not find container \"b770ac3dd989e5d53c0c84ec9e36ba801afacecce27c15eb247d01bf2bda419a\": container with ID starting with b770ac3dd989e5d53c0c84ec9e36ba801afacecce27c15eb247d01bf2bda419a not found: ID does not exist"
Sep 12 00:46:42.128768 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.128744    2567 scope.go:117] "RemoveContainer" containerID="cebee570b2eedcd77744cadbf65765550c057e427bc32b01b94a465bd704367f"
Sep 12 00:46:42.128981 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:46:42.128964    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cebee570b2eedcd77744cadbf65765550c057e427bc32b01b94a465bd704367f\": container with ID starting with cebee570b2eedcd77744cadbf65765550c057e427bc32b01b94a465bd704367f not found: ID does not exist" containerID="cebee570b2eedcd77744cadbf65765550c057e427bc32b01b94a465bd704367f"
Sep 12 00:46:42.129028 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.128987    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cebee570b2eedcd77744cadbf65765550c057e427bc32b01b94a465bd704367f"} err="failed to get container status \"cebee570b2eedcd77744cadbf65765550c057e427bc32b01b94a465bd704367f\": rpc error: code = NotFound desc = could not find container \"cebee570b2eedcd77744cadbf65765550c057e427bc32b01b94a465bd704367f\": container with ID starting with cebee570b2eedcd77744cadbf65765550c057e427bc32b01b94a465bd704367f not found: ID does not exist"
Sep 12 00:46:42.129028 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.129002    2567 scope.go:117] "RemoveContainer" containerID="8a8d8ce04eb8d044012820873915e510e3daef8fc737d4b64a84af7aa71d3a83"
Sep 12 00:46:42.129243 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:46:42.129225    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8d8ce04eb8d044012820873915e510e3daef8fc737d4b64a84af7aa71d3a83\": container with ID starting with 8a8d8ce04eb8d044012820873915e510e3daef8fc737d4b64a84af7aa71d3a83 not found: ID does not exist" containerID="8a8d8ce04eb8d044012820873915e510e3daef8fc737d4b64a84af7aa71d3a83"
Sep 12 00:46:42.129282 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.129247    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8d8ce04eb8d044012820873915e510e3daef8fc737d4b64a84af7aa71d3a83"} err="failed to get container status \"8a8d8ce04eb8d044012820873915e510e3daef8fc737d4b64a84af7aa71d3a83\": rpc error: code = NotFound desc = could not find container \"8a8d8ce04eb8d044012820873915e510e3daef8fc737d4b64a84af7aa71d3a83\": container with ID starting with 8a8d8ce04eb8d044012820873915e510e3daef8fc737d4b64a84af7aa71d3a83 not found: ID does not exist"
Sep 12 00:46:42.129282 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.129261    2567 scope.go:117] "RemoveContainer" containerID="10e7284f9bb61867e99d23fc0727b2de94d84db6146af11b3ff857da8a57eeb1"
Sep 12 00:46:42.129480 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:46:42.129467    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e7284f9bb61867e99d23fc0727b2de94d84db6146af11b3ff857da8a57eeb1\": container with ID starting with 10e7284f9bb61867e99d23fc0727b2de94d84db6146af11b3ff857da8a57eeb1 not found: ID does not exist" containerID="10e7284f9bb61867e99d23fc0727b2de94d84db6146af11b3ff857da8a57eeb1"
Sep 12 00:46:42.129514 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.129483    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e7284f9bb61867e99d23fc0727b2de94d84db6146af11b3ff857da8a57eeb1"} err="failed to get container status \"10e7284f9bb61867e99d23fc0727b2de94d84db6146af11b3ff857da8a57eeb1\": rpc error: code = NotFound desc = could not find container \"10e7284f9bb61867e99d23fc0727b2de94d84db6146af11b3ff857da8a57eeb1\": container with ID starting with 10e7284f9bb61867e99d23fc0727b2de94d84db6146af11b3ff857da8a57eeb1 not found: ID does not exist"
Sep 12 00:46:42.129514 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.129495    2567 scope.go:117] "RemoveContainer" containerID="655ed296c58473237243162066accf52e25ac1622c2516522b8a921bbaebc38b"
Sep 12 00:46:42.129724 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:46:42.129709    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655ed296c58473237243162066accf52e25ac1622c2516522b8a921bbaebc38b\": container with ID starting with 655ed296c58473237243162066accf52e25ac1622c2516522b8a921bbaebc38b not found: ID does not exist" containerID="655ed296c58473237243162066accf52e25ac1622c2516522b8a921bbaebc38b"
Sep 12 00:46:42.129778 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:42.129729    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655ed296c58473237243162066accf52e25ac1622c2516522b8a921bbaebc38b"} err="failed to get container status \"655ed296c58473237243162066accf52e25ac1622c2516522b8a921bbaebc38b\": rpc error: code = NotFound desc = could not find container \"655ed296c58473237243162066accf52e25ac1622c2516522b8a921bbaebc38b\": container with ID starting with 655ed296c58473237243162066accf52e25ac1622c2516522b8a921bbaebc38b not found: ID does not exist"
Sep 12 00:46:43.973276 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:43.973243    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" path="/var/lib/kubelet/pods/8a3606e9-aff1-42b4-ae29-1a577c25eb29/volumes"
Sep 12 00:46:44.495188 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.495148    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 00:46:44.495500 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.495459    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="prometheus" containerID="cri-o://404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17" gracePeriod=600
Sep 12 00:46:44.495643 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.495498    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513" gracePeriod=600
Sep 12 00:46:44.495643 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.495502    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="thanos-sidecar" containerID="cri-o://a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c" gracePeriod=600
Sep 12 00:46:44.495643 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.495563    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="kube-rbac-proxy" containerID="cri-o://b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b" gracePeriod=600
Sep 12 00:46:44.495801 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.495532    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="kube-rbac-proxy-web" containerID="cri-o://fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184" gracePeriod=600
Sep 12 00:46:44.495801 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.495752    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="config-reloader" containerID="cri-o://05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6" gracePeriod=600
Sep 12 00:46:44.748971 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.748881    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:44.881051 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.881013    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-grpc-tls\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.881051 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.881054    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-web-config\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.881293 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.881078    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-tls\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.881293 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.881124    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb2247e-8a33-46fd-8006-b12170d71f1e-config-out\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.881293 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.881153    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-kubelet-serving-ca-bundle\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.881293 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.881178    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-1\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.881293 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.881199    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-metrics-client-ca\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.881293 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.881218    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb2247e-8a33-46fd-8006-b12170d71f1e-tls-assets\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.881656 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.881581    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:46:44.881656 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.881643    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-metrics-client-certs\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.881841 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.881790    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:46:44.881841 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.881780    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-2\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.881841 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.881863    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.882150 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.881905    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-config\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.882150 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.882036    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.882257 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.882156    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:46:44.882352 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.882327    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:46:44.883489 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.882641    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-kube-rbac-proxy\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.883489 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.882687    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-db\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.883489 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.882721    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-0\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.883489 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.882749    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-trusted-ca-bundle\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.883489 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.882793    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg2j8\" (UniqueName: \"kubernetes.io/projected/ccb2247e-8a33-46fd-8006-b12170d71f1e-kube-api-access-bg2j8\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.883489 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.882818    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-serving-certs-ca-bundle\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.883489 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.882849    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-thanos-prometheus-http-client-file\") pod \"ccb2247e-8a33-46fd-8006-b12170d71f1e\" (UID: \"ccb2247e-8a33-46fd-8006-b12170d71f1e\") "
Sep 12 00:46:44.884074 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.883636    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:46:44.884074 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.884010    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.884074 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.884034    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.884074 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.884052    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-metrics-client-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.884074 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.884069    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.884319 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.884084    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.885051 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.884706    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:46:44.885051 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.884924    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:44.885051 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.884941    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:46:44.885429 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.885045    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb2247e-8a33-46fd-8006-b12170d71f1e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:46:44.885429 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.885046    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:44.885429 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.885114    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:44.885429 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.885246    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:44.885660 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.885488    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb2247e-8a33-46fd-8006-b12170d71f1e-config-out" (OuterVolumeSpecName: "config-out") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:44.885938 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.885902    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:44.886478 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.886452    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:44.886478 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.886466    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-config" (OuterVolumeSpecName: "config") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:44.887064 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.887040    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:44.887376 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.887355    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:44.887518 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.887499    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb2247e-8a33-46fd-8006-b12170d71f1e-kube-api-access-bg2j8" (OuterVolumeSpecName: "kube-api-access-bg2j8") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "kube-api-access-bg2j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:46:44.895931 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.895906    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-web-config" (OuterVolumeSpecName: "web-config") pod "ccb2247e-8a33-46fd-8006-b12170d71f1e" (UID: "ccb2247e-8a33-46fd-8006-b12170d71f1e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:44.984623 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984563    2567 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-metrics-client-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.984623 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984613    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.984623 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984627    2567 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-config\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.984623 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984639    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.985155 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984651    2567 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-kube-rbac-proxy\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.985155 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984665    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-db\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.985155 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984676    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.985155 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984688    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg2j8\" (UniqueName: \"kubernetes.io/projected/ccb2247e-8a33-46fd-8006-b12170d71f1e-kube-api-access-bg2j8\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.985155 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984699    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb2247e-8a33-46fd-8006-b12170d71f1e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.985155 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984713    2567 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.985155 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984727    2567 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-grpc-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.985155 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984740    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-web-config\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.985155 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984754    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb2247e-8a33-46fd-8006-b12170d71f1e-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.985155 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984767    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb2247e-8a33-46fd-8006-b12170d71f1e-config-out\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:44.985155 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:44.984782    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb2247e-8a33-46fd-8006-b12170d71f1e-tls-assets\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:46:45.094062 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.093975    2567 generic.go:356] "Generic (PLEG): container finished" podID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerID="44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513" exitCode=0
Sep 12 00:46:45.094062 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.093997    2567 generic.go:356] "Generic (PLEG): container finished" podID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerID="b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b" exitCode=0
Sep 12 00:46:45.094062 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.094003    2567 generic.go:356] "Generic (PLEG): container finished" podID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerID="fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184" exitCode=0
Sep 12 00:46:45.094062 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.094008    2567 generic.go:356] "Generic (PLEG): container finished" podID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerID="a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c" exitCode=0
Sep 12 00:46:45.094062 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.094013    2567 generic.go:356] "Generic (PLEG): container finished" podID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerID="05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6" exitCode=0
Sep 12 00:46:45.094062 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.094019    2567 generic.go:356] "Generic (PLEG): container finished" podID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerID="404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17" exitCode=0
Sep 12 00:46:45.094062 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.094064    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerDied","Data":"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"}
Sep 12 00:46:45.094538 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.094111    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.094538 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.094122    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerDied","Data":"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"}
Sep 12 00:46:45.094538 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.094151    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerDied","Data":"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"}
Sep 12 00:46:45.094538 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.094176    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerDied","Data":"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"}
Sep 12 00:46:45.094538 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.094196    2567 scope.go:117] "RemoveContainer" containerID="44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"
Sep 12 00:46:45.094538 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.094201    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerDied","Data":"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"}
Sep 12 00:46:45.094538 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.094300    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerDied","Data":"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"}
Sep 12 00:46:45.094538 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.094329    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb2247e-8a33-46fd-8006-b12170d71f1e","Type":"ContainerDied","Data":"de43306b473e3b1784a6749b4722627c0059f0e68bb53a364ce3e711feb825ab"}
Sep 12 00:46:45.103033 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.103007    2567 scope.go:117] "RemoveContainer" containerID="b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"
Sep 12 00:46:45.110299 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.110280    2567 scope.go:117] "RemoveContainer" containerID="fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"
Sep 12 00:46:45.118010 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.117982    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 00:46:45.118614 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.118593    2567 scope.go:117] "RemoveContainer" containerID="a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"
Sep 12 00:46:45.125658 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.125632    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 00:46:45.128654 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.128632    2567 scope.go:117] "RemoveContainer" containerID="05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"
Sep 12 00:46:45.136477 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.136455    2567 scope.go:117] "RemoveContainer" containerID="404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"
Sep 12 00:46:45.144350 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.144330    2567 scope.go:117] "RemoveContainer" containerID="e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"
Sep 12 00:46:45.151736 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.151717    2567 scope.go:117] "RemoveContainer" containerID="44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"
Sep 12 00:46:45.152074 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:46:45.152055    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513\": container with ID starting with 44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513 not found: ID does not exist" containerID="44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"
Sep 12 00:46:45.152154 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.152087    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"} err="failed to get container status \"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513\": rpc error: code = NotFound desc = could not find container \"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513\": container with ID starting with 44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513 not found: ID does not exist"
Sep 12 00:46:45.152154 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.152115    2567 scope.go:117] "RemoveContainer" containerID="b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"
Sep 12 00:46:45.152388 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:46:45.152368    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b\": container with ID starting with b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b not found: ID does not exist" containerID="b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"
Sep 12 00:46:45.152426 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.152395    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"} err="failed to get container status \"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b\": rpc error: code = NotFound desc = could not find container \"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b\": container with ID starting with b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b not found: ID does not exist"
Sep 12 00:46:45.152426 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.152414    2567 scope.go:117] "RemoveContainer" containerID="fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"
Sep 12 00:46:45.152640 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:46:45.152624    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184\": container with ID starting with fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184 not found: ID does not exist" containerID="fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"
Sep 12 00:46:45.152678 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.152645    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"} err="failed to get container status \"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184\": rpc error: code = NotFound desc = could not find container \"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184\": container with ID starting with fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184 not found: ID does not exist"
Sep 12 00:46:45.152678 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.152660    2567 scope.go:117] "RemoveContainer" containerID="a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"
Sep 12 00:46:45.152872 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:46:45.152854    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c\": container with ID starting with a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c not found: ID does not exist" containerID="a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"
Sep 12 00:46:45.152932 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.152879    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"} err="failed to get container status \"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c\": rpc error: code = NotFound desc = could not find container \"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c\": container with ID starting with a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c not found: ID does not exist"
Sep 12 00:46:45.152932 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.152901    2567 scope.go:117] "RemoveContainer" containerID="05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"
Sep 12 00:46:45.153387 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:46:45.153372    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6\": container with ID starting with 05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6 not found: ID does not exist" containerID="05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"
Sep 12 00:46:45.153436 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.153392    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"} err="failed to get container status \"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6\": rpc error: code = NotFound desc = could not find container \"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6\": container with ID starting with 05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6 not found: ID does not exist"
Sep 12 00:46:45.153436 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.153404    2567 scope.go:117] "RemoveContainer" containerID="404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"
Sep 12 00:46:45.153623 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:46:45.153607    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17\": container with ID starting with 404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17 not found: ID does not exist" containerID="404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"
Sep 12 00:46:45.153662 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.153626    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"} err="failed to get container status \"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17\": rpc error: code = NotFound desc = could not find container \"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17\": container with ID starting with 404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17 not found: ID does not exist"
Sep 12 00:46:45.153662 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.153638    2567 scope.go:117] "RemoveContainer" containerID="e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"
Sep 12 00:46:45.153856 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:46:45.153839    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0\": container with ID starting with e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0 not found: ID does not exist" containerID="e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"
Sep 12 00:46:45.153890 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.153860    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"} err="failed to get container status \"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0\": rpc error: code = NotFound desc = could not find container \"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0\": container with ID starting with e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0 not found: ID does not exist"
Sep 12 00:46:45.153890 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.153876    2567 scope.go:117] "RemoveContainer" containerID="44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"
Sep 12 00:46:45.154111 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.154096    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"} err="failed to get container status \"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513\": rpc error: code = NotFound desc = could not find container \"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513\": container with ID starting with 44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513 not found: ID does not exist"
Sep 12 00:46:45.154160 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.154112    2567 scope.go:117] "RemoveContainer" containerID="b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"
Sep 12 00:46:45.154316 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.154299    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"} err="failed to get container status \"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b\": rpc error: code = NotFound desc = could not find container \"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b\": container with ID starting with b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b not found: ID does not exist"
Sep 12 00:46:45.154360 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.154317    2567 scope.go:117] "RemoveContainer" containerID="fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"
Sep 12 00:46:45.154575 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.154558    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"} err="failed to get container status \"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184\": rpc error: code = NotFound desc = could not find container \"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184\": container with ID starting with fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184 not found: ID does not exist"
Sep 12 00:46:45.154657 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.154577    2567 scope.go:117] "RemoveContainer" containerID="a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"
Sep 12 00:46:45.154844 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.154827    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"} err="failed to get container status \"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c\": rpc error: code = NotFound desc = could not find container \"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c\": container with ID starting with a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c not found: ID does not exist"
Sep 12 00:46:45.154898 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.154845    2567 scope.go:117] "RemoveContainer" containerID="05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"
Sep 12 00:46:45.155076 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.155058    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"} err="failed to get container status \"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6\": rpc error: code = NotFound desc = could not find container \"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6\": container with ID starting with 05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6 not found: ID does not exist"
Sep 12 00:46:45.155127 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.155077    2567 scope.go:117] "RemoveContainer" containerID="404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"
Sep 12 00:46:45.155313 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.155292    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"} err="failed to get container status \"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17\": rpc error: code = NotFound desc = could not find container \"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17\": container with ID starting with 404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17 not found: ID does not exist"
Sep 12 00:46:45.155398 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.155314    2567 scope.go:117] "RemoveContainer" containerID="e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"
Sep 12 00:46:45.155669 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.155634    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"} err="failed to get container status \"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0\": rpc error: code = NotFound desc = could not find container \"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0\": container with ID starting with e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0 not found: ID does not exist"
Sep 12 00:46:45.155669 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.155668    2567 scope.go:117] "RemoveContainer" containerID="44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"
Sep 12 00:46:45.155998 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.155974    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"} err="failed to get container status \"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513\": rpc error: code = NotFound desc = could not find container \"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513\": container with ID starting with 44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513 not found: ID does not exist"
Sep 12 00:46:45.156079 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.156000    2567 scope.go:117] "RemoveContainer" containerID="b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"
Sep 12 00:46:45.156300 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.156266    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"} err="failed to get container status \"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b\": rpc error: code = NotFound desc = could not find container \"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b\": container with ID starting with b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b not found: ID does not exist"
Sep 12 00:46:45.156388 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.156301    2567 scope.go:117] "RemoveContainer" containerID="fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"
Sep 12 00:46:45.156666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.156645    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"} err="failed to get container status \"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184\": rpc error: code = NotFound desc = could not find container \"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184\": container with ID starting with fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184 not found: ID does not exist"
Sep 12 00:46:45.156666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.156663    2567 scope.go:117] "RemoveContainer" containerID="a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"
Sep 12 00:46:45.156909 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.156886    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"} err="failed to get container status \"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c\": rpc error: code = NotFound desc = could not find container \"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c\": container with ID starting with a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c not found: ID does not exist"
Sep 12 00:46:45.156997 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.156909    2567 scope.go:117] "RemoveContainer" containerID="05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"
Sep 12 00:46:45.157146 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157126    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"} err="failed to get container status \"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6\": rpc error: code = NotFound desc = could not find container \"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6\": container with ID starting with 05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6 not found: ID does not exist"
Sep 12 00:46:45.157207 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157146    2567 scope.go:117] "RemoveContainer" containerID="404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"
Sep 12 00:46:45.157386 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157368    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"} err="failed to get container status \"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17\": rpc error: code = NotFound desc = could not find container \"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17\": container with ID starting with 404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17 not found: ID does not exist"
Sep 12 00:46:45.157442 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157396    2567 scope.go:117] "RemoveContainer" containerID="e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"
Sep 12 00:46:45.157551 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157533    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 00:46:45.157633 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157609    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"} err="failed to get container status \"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0\": rpc error: code = NotFound desc = could not find container \"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0\": container with ID starting with e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0 not found: ID does not exist"
Sep 12 00:46:45.157695 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157635    2567 scope.go:117] "RemoveContainer" containerID="44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"
Sep 12 00:46:45.157846 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157830    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"} err="failed to get container status \"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513\": rpc error: code = NotFound desc = could not find container \"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513\": container with ID starting with 44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513 not found: ID does not exist"
Sep 12 00:46:45.157905 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157845    2567 scope.go:117] "RemoveContainer" containerID="b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"
Sep 12 00:46:45.157905 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157874    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="alertmanager"
Sep 12 00:46:45.157905 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157891    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="alertmanager"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157906    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="kube-rbac-proxy"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157914    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="kube-rbac-proxy"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157933    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="config-reloader"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157941    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="config-reloader"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157970    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="prometheus"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157978    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="prometheus"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157987    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="kube-rbac-proxy-metric"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.157996    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="kube-rbac-proxy-metric"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158004    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="kube-rbac-proxy-thanos"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158012    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="kube-rbac-proxy-thanos"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158030    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="thanos-sidecar"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158040    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="thanos-sidecar"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158050    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c654414-0339-4096-a9d1-fe3c1c609ec3" containerName="console"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158059    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c654414-0339-4096-a9d1-fe3c1c609ec3" containerName="console"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158068    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="prom-label-proxy"
Sep 12 00:46:45.158070 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158069    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"} err="failed to get container status \"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b\": rpc error: code = NotFound desc = could not find container \"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b\": container with ID starting with b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b not found: ID does not exist"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158085    2567 scope.go:117] "RemoveContainer" containerID="fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158077    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="prom-label-proxy"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158135    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="config-reloader"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158143    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="config-reloader"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158154    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="init-config-reloader"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158160    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="init-config-reloader"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158167    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="init-config-reloader"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158173    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="init-config-reloader"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158182    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="kube-rbac-proxy-web"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158188    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="kube-rbac-proxy-web"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158199    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="kube-rbac-proxy"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158207    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="kube-rbac-proxy"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158217    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="kube-rbac-proxy-web"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158226    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="kube-rbac-proxy-web"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158318    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="kube-rbac-proxy"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158337    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="kube-rbac-proxy-thanos"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158311    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"} err="failed to get container status \"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184\": rpc error: code = NotFound desc = could not find container \"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184\": container with ID starting with fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184 not found: ID does not exist"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158357    2567 scope.go:117] "RemoveContainer" containerID="a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158348    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c654414-0339-4096-a9d1-fe3c1c609ec3" containerName="console"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158399    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="prometheus"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158409    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="alertmanager"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158418    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="prom-label-proxy"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158427    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="kube-rbac-proxy"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158435    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="thanos-sidecar"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158441    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="config-reloader"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158446    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="kube-rbac-proxy-web"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158452    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="config-reloader"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158458    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" containerName="kube-rbac-proxy-web"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158463    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3606e9-aff1-42b4-ae29-1a577c25eb29" containerName="kube-rbac-proxy-metric"
Sep 12 00:46:45.158582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158583    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"} err="failed to get container status \"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c\": rpc error: code = NotFound desc = could not find container \"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c\": container with ID starting with a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c not found: ID does not exist"
Sep 12 00:46:45.159681 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158603    2567 scope.go:117] "RemoveContainer" containerID="05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"
Sep 12 00:46:45.159681 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158849    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"} err="failed to get container status \"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6\": rpc error: code = NotFound desc = could not find container \"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6\": container with ID starting with 05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6 not found: ID does not exist"
Sep 12 00:46:45.159681 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.158862    2567 scope.go:117] "RemoveContainer" containerID="404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"
Sep 12 00:46:45.159681 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.159073    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"} err="failed to get container status \"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17\": rpc error: code = NotFound desc = could not find container \"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17\": container with ID starting with 404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17 not found: ID does not exist"
Sep 12 00:46:45.159681 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.159088    2567 scope.go:117] "RemoveContainer" containerID="e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"
Sep 12 00:46:45.159681 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.159304    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"} err="failed to get container status \"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0\": rpc error: code = NotFound desc = could not find container \"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0\": container with ID starting with e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0 not found: ID does not exist"
Sep 12 00:46:45.159681 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.159317    2567 scope.go:117] "RemoveContainer" containerID="44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"
Sep 12 00:46:45.159681 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.159497    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"} err="failed to get container status \"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513\": rpc error: code = NotFound desc = could not find container \"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513\": container with ID starting with 44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513 not found: ID does not exist"
Sep 12 00:46:45.159681 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.159510    2567 scope.go:117] "RemoveContainer" containerID="b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"
Sep 12 00:46:45.160022 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.159735    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"} err="failed to get container status \"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b\": rpc error: code = NotFound desc = could not find container \"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b\": container with ID starting with b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b not found: ID does not exist"
Sep 12 00:46:45.160022 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.159756    2567 scope.go:117] "RemoveContainer" containerID="fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"
Sep 12 00:46:45.160022 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.160003    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"} err="failed to get container status \"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184\": rpc error: code = NotFound desc = could not find container \"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184\": container with ID starting with fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184 not found: ID does not exist"
Sep 12 00:46:45.160022 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.160021    2567 scope.go:117] "RemoveContainer" containerID="a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"
Sep 12 00:46:45.160278 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.160260    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"} err="failed to get container status \"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c\": rpc error: code = NotFound desc = could not find container \"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c\": container with ID starting with a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c not found: ID does not exist"
Sep 12 00:46:45.160323 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.160278    2567 scope.go:117] "RemoveContainer" containerID="05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"
Sep 12 00:46:45.160493 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.160475    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"} err="failed to get container status \"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6\": rpc error: code = NotFound desc = could not find container \"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6\": container with ID starting with 05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6 not found: ID does not exist"
Sep 12 00:46:45.160537 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.160494    2567 scope.go:117] "RemoveContainer" containerID="404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"
Sep 12 00:46:45.160726 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.160705    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"} err="failed to get container status \"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17\": rpc error: code = NotFound desc = could not find container \"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17\": container with ID starting with 404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17 not found: ID does not exist"
Sep 12 00:46:45.160773 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.160726    2567 scope.go:117] "RemoveContainer" containerID="e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"
Sep 12 00:46:45.160939 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.160924    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"} err="failed to get container status \"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0\": rpc error: code = NotFound desc = could not find container \"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0\": container with ID starting with e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0 not found: ID does not exist"
Sep 12 00:46:45.161009 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.160941    2567 scope.go:117] "RemoveContainer" containerID="44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"
Sep 12 00:46:45.161176 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.161160    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513"} err="failed to get container status \"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513\": rpc error: code = NotFound desc = could not find container \"44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513\": container with ID starting with 44dbba288499e62a90e37a7bbd2101f79da1c0b25486f092dfde4960c9096513 not found: ID does not exist"
Sep 12 00:46:45.161176 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.161174    2567 scope.go:117] "RemoveContainer" containerID="b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"
Sep 12 00:46:45.161390 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.161372    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b"} err="failed to get container status \"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b\": rpc error: code = NotFound desc = could not find container \"b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b\": container with ID starting with b15b423755798cbef57c3c5f108f9addd4a86118f9360af685a92f006299937b not found: ID does not exist"
Sep 12 00:46:45.161428 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.161391    2567 scope.go:117] "RemoveContainer" containerID="fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"
Sep 12 00:46:45.161581 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.161567    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184"} err="failed to get container status \"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184\": rpc error: code = NotFound desc = could not find container \"fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184\": container with ID starting with fa2f31df31186ce010add6004cd20ba02c4dcb087964e79431d3a8e1eaef9184 not found: ID does not exist"
Sep 12 00:46:45.161628 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.161580    2567 scope.go:117] "RemoveContainer" containerID="a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"
Sep 12 00:46:45.161807 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.161790    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c"} err="failed to get container status \"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c\": rpc error: code = NotFound desc = could not find container \"a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c\": container with ID starting with a65a491f40778925c54f84a4a3b878923b6075dd2773003a20e30540f87d7a7c not found: ID does not exist"
Sep 12 00:46:45.161807 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.161806    2567 scope.go:117] "RemoveContainer" containerID="05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"
Sep 12 00:46:45.162018 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.161998    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6"} err="failed to get container status \"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6\": rpc error: code = NotFound desc = could not find container \"05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6\": container with ID starting with 05a26e07438e09dce4a3819f9a52f6890f9c0fd36adb757d8b946d54970efca6 not found: ID does not exist"
Sep 12 00:46:45.162066 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.162019    2567 scope.go:117] "RemoveContainer" containerID="404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"
Sep 12 00:46:45.162248 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.162232    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17"} err="failed to get container status \"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17\": rpc error: code = NotFound desc = could not find container \"404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17\": container with ID starting with 404015e1f901f858e50d728e861d7bf731fb0bd3bc186357af400292ca237c17 not found: ID does not exist"
Sep 12 00:46:45.162248 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.162247    2567 scope.go:117] "RemoveContainer" containerID="e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"
Sep 12 00:46:45.162468 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.162453    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0"} err="failed to get container status \"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0\": rpc error: code = NotFound desc = could not find container \"e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0\": container with ID starting with e152cbf1540b5a38c36e89345717adde4f84237ac156490fc9726a0a8eb3cab0 not found: ID does not exist"
Sep 12 00:46:45.163959 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.163935    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.166573 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.166554    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 12 00:46:45.166657 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.166576    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 12 00:46:45.167420 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.167294    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 12 00:46:45.167420 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.167305    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jfm55\""
Sep 12 00:46:45.167420 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.167312    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 12 00:46:45.167420 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.167296    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 12 00:46:45.167420 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.167318    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 12 00:46:45.167420 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.167296    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 12 00:46:45.167420 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.167324    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8sdddgerbd5kh\""
Sep 12 00:46:45.167711 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.167626    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 12 00:46:45.168027 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.168010    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 12 00:46:45.168247 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.168231    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 12 00:46:45.168345 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.168330    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 12 00:46:45.168601 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.168586    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 12 00:46:45.171352 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.171333    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 12 00:46:45.175043 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.175023    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 12 00:46:45.185041 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.185015    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 00:46:45.287332 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287304    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287482 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287339    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287482 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287357    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287482 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287425    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287482 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287455    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287609 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287501    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287609 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287523    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287609 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287543    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb58q\" (UniqueName: \"kubernetes.io/projected/11db4521-b277-45e1-a609-7c48b808b13f-kube-api-access-lb58q\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287609 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287559    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287609 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287575    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287756 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287611    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11db4521-b277-45e1-a609-7c48b808b13f-config-out\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287756 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287643    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-web-config\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287756 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287659    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287756 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287674    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287756 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287735    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11db4521-b277-45e1-a609-7c48b808b13f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287901 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287771    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287901 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287791    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287901 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287814    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-config\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287901 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287841    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.287901 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.287865    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.388852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.388740    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-web-config\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.388852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.388798    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.388852 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.388815    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389145 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.388886    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11db4521-b277-45e1-a609-7c48b808b13f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389145 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.388935    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389145 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.388988    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389145 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389013    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-config\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389145 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389043    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389145 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389076    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389145 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389130    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389492 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389161    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389492 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389190    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389492 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389223    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389492 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389257    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389492 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389283    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389492 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389309    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389492 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389332    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb58q\" (UniqueName: \"kubernetes.io/projected/11db4521-b277-45e1-a609-7c48b808b13f-kube-api-access-lb58q\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389492 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389363    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389492 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389391    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389492 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389415    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11db4521-b277-45e1-a609-7c48b808b13f-config-out\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389492 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389468    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.389492 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389482    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.390271 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.389582    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.390271 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.390011    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.392265 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.392227    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.392398 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.392335    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-config\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.392715 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.392486    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.392715 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.392534    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11db4521-b277-45e1-a609-7c48b808b13f-config-out\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.392858 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.392738    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.392912 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.392855    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-web-config\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.393590 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.393232    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.393590 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.393330    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.393590 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.393394    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.393788 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.393639    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.394562 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.394536    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.394758 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.394739    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.394892 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.394874    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11db4521-b277-45e1-a609-7c48b808b13f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.395288 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.395271    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11db4521-b277-45e1-a609-7c48b808b13f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.395844 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.395827    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11db4521-b277-45e1-a609-7c48b808b13f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.400049 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.400029    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb58q\" (UniqueName: \"kubernetes.io/projected/11db4521-b277-45e1-a609-7c48b808b13f-kube-api-access-lb58q\") pod \"prometheus-k8s-0\" (UID: \"11db4521-b277-45e1-a609-7c48b808b13f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.476844 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.476805    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:46:45.759242 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.759214    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 00:46:45.761289 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:46:45.761246    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11db4521_b277_45e1_a609_7c48b808b13f.slice/crio-ca5108d3fca4fe86c48ccd89f94733fb75c31562f4d84e274b36921b3b806065 WatchSource:0}: Error finding container ca5108d3fca4fe86c48ccd89f94733fb75c31562f4d84e274b36921b3b806065: Status 404 returned error can't find the container with id ca5108d3fca4fe86c48ccd89f94733fb75c31562f4d84e274b36921b3b806065
Sep 12 00:46:45.971622 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:45.971589    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccb2247e-8a33-46fd-8006-b12170d71f1e" path="/var/lib/kubelet/pods/ccb2247e-8a33-46fd-8006-b12170d71f1e/volumes"
Sep 12 00:46:46.100634 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:46.100544    2567 generic.go:356] "Generic (PLEG): container finished" podID="11db4521-b277-45e1-a609-7c48b808b13f" containerID="0949e09a4c3399375eb95d2c646f07b281e2149e6415c9fffbe1e17173a69eb0" exitCode=0
Sep 12 00:46:46.100634 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:46.100593    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11db4521-b277-45e1-a609-7c48b808b13f","Type":"ContainerDied","Data":"0949e09a4c3399375eb95d2c646f07b281e2149e6415c9fffbe1e17173a69eb0"}
Sep 12 00:46:46.101076 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:46.100644    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11db4521-b277-45e1-a609-7c48b808b13f","Type":"ContainerStarted","Data":"ca5108d3fca4fe86c48ccd89f94733fb75c31562f4d84e274b36921b3b806065"}
Sep 12 00:46:47.108517 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:47.108480    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11db4521-b277-45e1-a609-7c48b808b13f","Type":"ContainerStarted","Data":"3ea49a65fa4919795c3baca1d897e7c9abec4e3184f90382ba8c23064a9a7f9e"}
Sep 12 00:46:47.108889 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:47.108551    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11db4521-b277-45e1-a609-7c48b808b13f","Type":"ContainerStarted","Data":"4258f51d80438cd4782705eb0ff791f27df8502f1e6a8cd6b9c641f18946a99f"}
Sep 12 00:46:47.108889 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:47.108583    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11db4521-b277-45e1-a609-7c48b808b13f","Type":"ContainerStarted","Data":"527be29811561a0515c1d136be0bef2c52bf316f06296a7455b98e9f219b7bfe"}
Sep 12 00:46:47.108889 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:47.108612    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11db4521-b277-45e1-a609-7c48b808b13f","Type":"ContainerStarted","Data":"5125f313dba90c56711df39f253006bc0499285ff43fc66a1301725f9cb6cc4c"}
Sep 12 00:46:47.108889 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:47.108635    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11db4521-b277-45e1-a609-7c48b808b13f","Type":"ContainerStarted","Data":"23025a3baa6915128d973e7d3a9ec56e10f923b3eb2670903243a793c595ac08"}
Sep 12 00:46:47.108889 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:47.108657    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11db4521-b277-45e1-a609-7c48b808b13f","Type":"ContainerStarted","Data":"85fd379e62a457934c1a641e5802f20034dbeb606278572d0332ce403cc58e23"}
Sep 12 00:46:47.140442 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:47.140377    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.122091802 podStartE2EDuration="2.140355092s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.26329ms" isStatefulPod=false podCreationTimestamp="2026-09-12 00:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:46:47.139611227 +0000 UTC m=+211.700062894" watchObservedRunningTime="2026-09-12 00:46:47.140355092 +0000 UTC m=+211.700806754"
Sep 12 00:46:47.937177 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:47.937128    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:48.003478 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:48.003446    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:50.477245 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:46:50.477205    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:47:01.645479 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:47:01.645439    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:01.743548 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:47:01.743511    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:03.701242 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:47:03.701209    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:03.756572 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:47:03.756530    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:14.881480 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:47:14.881446    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:14.942608 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:47:14.942576    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:15.009337 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:47:15.009311    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:40.176676 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:47:40.176576    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:40.275959 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:47:40.275912    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:45.477460 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:47:45.477411    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:47:45.493582 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:47:45.493555    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:47:46.354176 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:47:46.354150    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 00:48:02.107668 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:02.107634    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:03.474404 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:03.474357    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:05.722119 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:05.722087    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:05.819144 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:05.819109    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:07.405349 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:07.405323    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:07.555191 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:07.555155    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:07.965560 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:07.965530    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:08.065409 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:08.065382    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:10.172779 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:10.172748    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:10.382476 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:10.382446    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:12.725078 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:12.725046    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:12.788662 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:12.788629    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:15.199298 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:15.199260    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:15.258839 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:15.258808    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:15.835730 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:15.835701    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 00:48:15.842376 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:15.842352    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 00:48:15.847856 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:15.847829    2567 kubelet.go:1648] "Image garbage collection succeeded"
Sep 12 00:48:18.369199 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:18.369163    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:21.013810 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:21.013773    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:21.109227 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:21.109201    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:22.689827 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:22.689787    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:22.790739 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:22.790706    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:25.772773 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:25.772743    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:25.837666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:25.837636    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:27.791901 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:27.791864    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:37.861213 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:37.861183    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:41.371358 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:41.371326    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:41.483393 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:41.483354    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:41.988811 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:41.988770    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:42.090080 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.090024    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:42.148581 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.148542    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"]
Sep 12 00:48:42.151870 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.151847    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:42.158274 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.158097    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 12 00:48:42.158417 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.158396    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 12 00:48:42.158470 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.158454    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 12 00:48:42.159164 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.159148    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 12 00:48:42.159164 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.159158    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-dhlkx\""
Sep 12 00:48:42.159306 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.159191    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:48:42.166217 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.166197    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"]
Sep 12 00:48:42.201702 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.201660    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:42.214830 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.214798    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c090878f-bc8f-40a3-af92-acce749fe642-manager-config\") pod \"lws-controller-manager-6ccd6666c9-vzvpv\" (UID: \"c090878f-bc8f-40a3-af92-acce749fe642\") " pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:42.214942 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.214849    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fbk4\" (UniqueName: \"kubernetes.io/projected/c090878f-bc8f-40a3-af92-acce749fe642-kube-api-access-4fbk4\") pod \"lws-controller-manager-6ccd6666c9-vzvpv\" (UID: \"c090878f-bc8f-40a3-af92-acce749fe642\") " pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:42.214942 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.214876    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c090878f-bc8f-40a3-af92-acce749fe642-metrics-cert\") pod \"lws-controller-manager-6ccd6666c9-vzvpv\" (UID: \"c090878f-bc8f-40a3-af92-acce749fe642\") " pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:42.214942 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.214901    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c090878f-bc8f-40a3-af92-acce749fe642-cert\") pod \"lws-controller-manager-6ccd6666c9-vzvpv\" (UID: \"c090878f-bc8f-40a3-af92-acce749fe642\") " pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:42.316014 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.315903    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c090878f-bc8f-40a3-af92-acce749fe642-manager-config\") pod \"lws-controller-manager-6ccd6666c9-vzvpv\" (UID: \"c090878f-bc8f-40a3-af92-acce749fe642\") " pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:42.316014 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.315989    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fbk4\" (UniqueName: \"kubernetes.io/projected/c090878f-bc8f-40a3-af92-acce749fe642-kube-api-access-4fbk4\") pod \"lws-controller-manager-6ccd6666c9-vzvpv\" (UID: \"c090878f-bc8f-40a3-af92-acce749fe642\") " pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:42.316240 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.316019    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c090878f-bc8f-40a3-af92-acce749fe642-metrics-cert\") pod \"lws-controller-manager-6ccd6666c9-vzvpv\" (UID: \"c090878f-bc8f-40a3-af92-acce749fe642\") " pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:42.316240 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.316036    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c090878f-bc8f-40a3-af92-acce749fe642-cert\") pod \"lws-controller-manager-6ccd6666c9-vzvpv\" (UID: \"c090878f-bc8f-40a3-af92-acce749fe642\") " pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:42.316586 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.316558    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c090878f-bc8f-40a3-af92-acce749fe642-manager-config\") pod \"lws-controller-manager-6ccd6666c9-vzvpv\" (UID: \"c090878f-bc8f-40a3-af92-acce749fe642\") " pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:42.318586 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.318566    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c090878f-bc8f-40a3-af92-acce749fe642-metrics-cert\") pod \"lws-controller-manager-6ccd6666c9-vzvpv\" (UID: \"c090878f-bc8f-40a3-af92-acce749fe642\") " pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:42.318678 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.318606    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c090878f-bc8f-40a3-af92-acce749fe642-cert\") pod \"lws-controller-manager-6ccd6666c9-vzvpv\" (UID: \"c090878f-bc8f-40a3-af92-acce749fe642\") " pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:42.348354 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.348321    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fbk4\" (UniqueName: \"kubernetes.io/projected/c090878f-bc8f-40a3-af92-acce749fe642-kube-api-access-4fbk4\") pod \"lws-controller-manager-6ccd6666c9-vzvpv\" (UID: \"c090878f-bc8f-40a3-af92-acce749fe642\") " pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:42.374607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.374572    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:42.460599 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.460560    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:42.468863 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.468824    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:42.523381 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.523343    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"]
Sep 12 00:48:42.527633 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.527608    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.530312 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.530294    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 12 00:48:42.532162 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.531700    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 12 00:48:42.532162 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.531786    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 12 00:48:42.532678 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.532629    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 12 00:48:42.533330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.533024    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lr2d4\""
Sep 12 00:48:42.544691 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.544671    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"]
Sep 12 00:48:42.618161 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.618065    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fc585fd3-4864-46bf-801d-7a598e8b8b14-manifests\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.618161 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.618103    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fc585fd3-4864-46bf-801d-7a598e8b8b14-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.618161 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.618130    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc585fd3-4864-46bf-801d-7a598e8b8b14-webhook-cert\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.618392 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.618235    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqbp2\" (UniqueName: \"kubernetes.io/projected/fc585fd3-4864-46bf-801d-7a598e8b8b14-kube-api-access-hqbp2\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.618392 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.618285    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc585fd3-4864-46bf-801d-7a598e8b8b14-tmp\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.683429 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.683389    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:42.718708 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.718676    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fc585fd3-4864-46bf-801d-7a598e8b8b14-manifests\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.718708 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.718713    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fc585fd3-4864-46bf-801d-7a598e8b8b14-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.718922 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.718733    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc585fd3-4864-46bf-801d-7a598e8b8b14-webhook-cert\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.718922 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.718769    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqbp2\" (UniqueName: \"kubernetes.io/projected/fc585fd3-4864-46bf-801d-7a598e8b8b14-kube-api-access-hqbp2\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.718922 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.718797    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc585fd3-4864-46bf-801d-7a598e8b8b14-tmp\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.719170 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.719150    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fc585fd3-4864-46bf-801d-7a598e8b8b14-manifests\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.721198 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.721162    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc585fd3-4864-46bf-801d-7a598e8b8b14-tmp\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.721334 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.721320    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc585fd3-4864-46bf-801d-7a598e8b8b14-webhook-cert\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.721435 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.721417    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fc585fd3-4864-46bf-801d-7a598e8b8b14-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.735847 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.735823    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqbp2\" (UniqueName: \"kubernetes.io/projected/fc585fd3-4864-46bf-801d-7a598e8b8b14-kube-api-access-hqbp2\") pod \"opendatahub-operator-controller-manager-694bbb96f5-gnqwd\" (UID: \"fc585fd3-4864-46bf-801d-7a598e8b8b14\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.748874 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.748852    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"]
Sep 12 00:48:42.750730 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:48:42.750702    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc090878f_bc8f_40a3_af92_acce749fe642.slice/crio-16eb186e82386929343dbaeb81a7d5f58bddab94126999f2011a35c504bbe2fa WatchSource:0}: Error finding container 16eb186e82386929343dbaeb81a7d5f58bddab94126999f2011a35c504bbe2fa: Status 404 returned error can't find the container with id 16eb186e82386929343dbaeb81a7d5f58bddab94126999f2011a35c504bbe2fa
Sep 12 00:48:42.752558 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.752542    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:48:42.804434 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.804385    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:42.839721 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.839681    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:42.880652 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.880538    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:42.984053 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:42.984014    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:43.096722 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:43.096690    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:43.122275 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:43.122245    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"]
Sep 12 00:48:43.126380 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:48:43.126350    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc585fd3_4864_46bf_801d_7a598e8b8b14.slice/crio-b104cb3c18d657ff286636f1f2487d2ae2e9c44ea5bf885d7150ec395d216c52 WatchSource:0}: Error finding container b104cb3c18d657ff286636f1f2487d2ae2e9c44ea5bf885d7150ec395d216c52: Status 404 returned error can't find the container with id b104cb3c18d657ff286636f1f2487d2ae2e9c44ea5bf885d7150ec395d216c52
Sep 12 00:48:43.566072 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:43.566005    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv" event={"ID":"c090878f-bc8f-40a3-af92-acce749fe642","Type":"ContainerStarted","Data":"16eb186e82386929343dbaeb81a7d5f58bddab94126999f2011a35c504bbe2fa"}
Sep 12 00:48:43.567758 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:43.567715    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd" event={"ID":"fc585fd3-4864-46bf-801d-7a598e8b8b14","Type":"ContainerStarted","Data":"b104cb3c18d657ff286636f1f2487d2ae2e9c44ea5bf885d7150ec395d216c52"}
Sep 12 00:48:46.592306 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:46.592264    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd" event={"ID":"fc585fd3-4864-46bf-801d-7a598e8b8b14","Type":"ContainerStarted","Data":"c39031c980504d68db2b509e5399f76b2b9ebf7aa0d935ed2de7c36c9fc1430b"}
Sep 12 00:48:47.598012 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:47.597969    2567 generic.go:356] "Generic (PLEG): container finished" podID="fc585fd3-4864-46bf-801d-7a598e8b8b14" containerID="c39031c980504d68db2b509e5399f76b2b9ebf7aa0d935ed2de7c36c9fc1430b" exitCode=0
Sep 12 00:48:47.598415 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:47.598047    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd" event={"ID":"fc585fd3-4864-46bf-801d-7a598e8b8b14","Type":"ContainerDied","Data":"c39031c980504d68db2b509e5399f76b2b9ebf7aa0d935ed2de7c36c9fc1430b"}
Sep 12 00:48:48.605470 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:48.605435    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd" event={"ID":"fc585fd3-4864-46bf-801d-7a598e8b8b14","Type":"ContainerStarted","Data":"22e9d88c7f1dbd5881e1c994964552be58e703842f2f16ee053cfa83cf4f4cf6"}
Sep 12 00:48:48.605853 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:48.605533    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:48:48.631550 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:48.631505    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd" podStartSLOduration=2.530384964 podStartE2EDuration="6.631489751s" totalImagesPullingTime="3.030139961s" totalInitContainerRuntime="1.070964826s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:48:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:48:48.630579842 +0000 UTC m=+333.191031530" watchObservedRunningTime="2026-09-12 00:48:48.631489751 +0000 UTC m=+333.191941419"
Sep 12 00:48:53.628209 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:53.628172    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv" event={"ID":"c090878f-bc8f-40a3-af92-acce749fe642","Type":"ContainerStarted","Data":"ad9439a64353c891cc388a2b083dcd01668d2d0d033f8c7f2d4925d22693c420"}
Sep 12 00:48:53.628626 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:53.628265    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:48:53.646468 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:53.646418    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv" podStartSLOduration=1.341411385 podStartE2EDuration="11.646404113s" totalImagesPullingTime="10.304992728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:48:53.645468622 +0000 UTC m=+338.205920310" watchObservedRunningTime="2026-09-12 00:48:53.646404113 +0000 UTC m=+338.206855780"
Sep 12 00:48:54.421299 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:54.421262    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:54.519980 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:54.519942    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:59.611999 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:48:59.611967    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-gnqwd"
Sep 12 00:49:01.796135 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:01.796103    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:02.368833 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.368802    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9"]
Sep 12 00:49:02.377596 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.377569    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9"
Sep 12 00:49:02.380052 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.380030    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 12 00:49:02.380182 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.380116    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 12 00:49:02.380182 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.380138    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 12 00:49:02.380840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.380818    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 12 00:49:02.380969 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.380924    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2gsws\""
Sep 12 00:49:02.391071 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.391049    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9"]
Sep 12 00:49:02.496077 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.496029    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z9gw\" (UniqueName: \"kubernetes.io/projected/268927fe-2fda-4277-98ed-679fea20c9ca-kube-api-access-4z9gw\") pod \"kube-auth-proxy-5f75b6b767-xz9v9\" (UID: \"268927fe-2fda-4277-98ed-679fea20c9ca\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9"
Sep 12 00:49:02.496077 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.496084    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/268927fe-2fda-4277-98ed-679fea20c9ca-tls-certs\") pod \"kube-auth-proxy-5f75b6b767-xz9v9\" (UID: \"268927fe-2fda-4277-98ed-679fea20c9ca\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9"
Sep 12 00:49:02.496291 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.496165    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/268927fe-2fda-4277-98ed-679fea20c9ca-tmp\") pod \"kube-auth-proxy-5f75b6b767-xz9v9\" (UID: \"268927fe-2fda-4277-98ed-679fea20c9ca\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9"
Sep 12 00:49:02.597053 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.597015    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/268927fe-2fda-4277-98ed-679fea20c9ca-tmp\") pod \"kube-auth-proxy-5f75b6b767-xz9v9\" (UID: \"268927fe-2fda-4277-98ed-679fea20c9ca\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9"
Sep 12 00:49:02.597246 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.597087    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z9gw\" (UniqueName: \"kubernetes.io/projected/268927fe-2fda-4277-98ed-679fea20c9ca-kube-api-access-4z9gw\") pod \"kube-auth-proxy-5f75b6b767-xz9v9\" (UID: \"268927fe-2fda-4277-98ed-679fea20c9ca\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9"
Sep 12 00:49:02.597246 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.597125    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/268927fe-2fda-4277-98ed-679fea20c9ca-tls-certs\") pod \"kube-auth-proxy-5f75b6b767-xz9v9\" (UID: \"268927fe-2fda-4277-98ed-679fea20c9ca\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9"
Sep 12 00:49:02.599395 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.599368    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/268927fe-2fda-4277-98ed-679fea20c9ca-tmp\") pod \"kube-auth-proxy-5f75b6b767-xz9v9\" (UID: \"268927fe-2fda-4277-98ed-679fea20c9ca\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9"
Sep 12 00:49:02.599579 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.599563    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/268927fe-2fda-4277-98ed-679fea20c9ca-tls-certs\") pod \"kube-auth-proxy-5f75b6b767-xz9v9\" (UID: \"268927fe-2fda-4277-98ed-679fea20c9ca\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9"
Sep 12 00:49:02.635721 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.635660    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z9gw\" (UniqueName: \"kubernetes.io/projected/268927fe-2fda-4277-98ed-679fea20c9ca-kube-api-access-4z9gw\") pod \"kube-auth-proxy-5f75b6b767-xz9v9\" (UID: \"268927fe-2fda-4277-98ed-679fea20c9ca\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9"
Sep 12 00:49:02.687489 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.687458    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9"
Sep 12 00:49:02.961476 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:02.961451    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9"]
Sep 12 00:49:02.964002 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:49:02.963976    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268927fe_2fda_4277_98ed_679fea20c9ca.slice/crio-0ec31176d65ecf5c0cd2ad86a2c7d5352ab6c2a649c1d704518c7686b620da7e WatchSource:0}: Error finding container 0ec31176d65ecf5c0cd2ad86a2c7d5352ab6c2a649c1d704518c7686b620da7e: Status 404 returned error can't find the container with id 0ec31176d65ecf5c0cd2ad86a2c7d5352ab6c2a649c1d704518c7686b620da7e
Sep 12 00:49:03.102498 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:03.102465    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:03.679114 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:03.679074    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9" event={"ID":"268927fe-2fda-4277-98ed-679fea20c9ca","Type":"ContainerStarted","Data":"0ec31176d65ecf5c0cd2ad86a2c7d5352ab6c2a649c1d704518c7686b620da7e"}
Sep 12 00:49:03.900912 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:03.900875    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:04.040129 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:04.040046    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:04.636328 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:04.636296    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6ccd6666c9-vzvpv"
Sep 12 00:49:05.117789 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:05.117750    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:06.415613 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:06.415568    2567 prober.go:120] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-121.ec2.internal" podUID="ad9778864c1c592078495eb7196d09e3" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Sep 12 00:49:06.692909 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:06.692800    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9" event={"ID":"268927fe-2fda-4277-98ed-679fea20c9ca","Type":"ContainerStarted","Data":"84a28f8f1debe2cb1928fc53c7dc05772b76240d17ecae3391dfadb41523ea51"}
Sep 12 00:49:06.710422 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:06.710343    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f75b6b767-xz9v9" podStartSLOduration=1.925015629 podStartE2EDuration="4.7103284s" totalImagesPullingTime="2.785312771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:49:06.709030828 +0000 UTC m=+351.269482526" watchObservedRunningTime="2026-09-12 00:49:06.7103284 +0000 UTC m=+351.270780122"
Sep 12 00:49:08.910119 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:08.910089    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:08.993419 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:08.993380    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:09.265000 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:09.264875    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:09.392627 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:09.392598    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:09.492622 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:09.492580    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:12.873680 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:12.873652    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:13.022892 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:13.022847    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:13.123868 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:13.123773    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:13.231377 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:13.231339    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:13.505458 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:13.505414    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:13.742823 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:13.742795    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:22.575761 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:22.575714    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:22.705739 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:22.705707    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:22.903286 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:22.903200    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:22.996262 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:22.996234    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:23.998597 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:23.998565    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:24.197246 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:24.197212    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:30.143243 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:30.143210    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:30.294575 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:49:30.294548    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:13.944533 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:13.944498    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:14.040147 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:14.040117    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:28.904676 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:28.904645    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:29.700122 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:29.700093    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:32.802109 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:32.802074    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:32.908627 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:32.908598    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:33.005578 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:33.005536    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:33.981302 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:33.981266    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:34.084385 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:34.084349    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:34.192452 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:34.192409    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:34.731826 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:34.731791    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:34.837133 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:34.837100    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:35.353904 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:35.353866    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:35.571002 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:35.570970    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:37.301658 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:37.301583    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:37.713500 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:37.713468    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:38.128215 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:38.128137    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:38.404983 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:38.404879    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:38.512573 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:38.512547    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:42.515007 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:42.514977    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:42.618312 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:42.618286    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:43.133925 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:43.133870    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:43.229389 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:43.229355    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:45.074106 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:45.074075    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:45.280564 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:45.280535    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:45.380811 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:45.380736    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:45.535631 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:45.535598    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:45.636089 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:45.636022    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:48.495697 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.495661    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:48.508834 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.508806    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp"]
Sep 12 00:50:48.512132 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.512109    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp"
Sep 12 00:50:48.514772 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.514753    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 12 00:50:48.514908 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.514771    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 12 00:50:48.515043 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.515026    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 12 00:50:48.515816 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.515799    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-phdxx\""
Sep 12 00:50:48.515816 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.515807    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 12 00:50:48.529045 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.529016    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp"]
Sep 12 00:50:48.593667 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.593640    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:48.595875 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.595857    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpgz8\" (UniqueName: \"kubernetes.io/projected/6456a19b-fcb4-4955-aee2-ad30ccc7f4ca-kube-api-access-xpgz8\") pod \"kuadrant-console-plugin-fbdd4f445-m2sqp\" (UID: \"6456a19b-fcb4-4955-aee2-ad30ccc7f4ca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp"
Sep 12 00:50:48.595957 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.595903    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6456a19b-fcb4-4955-aee2-ad30ccc7f4ca-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-m2sqp\" (UID: \"6456a19b-fcb4-4955-aee2-ad30ccc7f4ca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp"
Sep 12 00:50:48.596016 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.596002    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6456a19b-fcb4-4955-aee2-ad30ccc7f4ca-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-m2sqp\" (UID: \"6456a19b-fcb4-4955-aee2-ad30ccc7f4ca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp"
Sep 12 00:50:48.697271 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.697235    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6456a19b-fcb4-4955-aee2-ad30ccc7f4ca-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-m2sqp\" (UID: \"6456a19b-fcb4-4955-aee2-ad30ccc7f4ca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp"
Sep 12 00:50:48.697439 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.697299    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpgz8\" (UniqueName: \"kubernetes.io/projected/6456a19b-fcb4-4955-aee2-ad30ccc7f4ca-kube-api-access-xpgz8\") pod \"kuadrant-console-plugin-fbdd4f445-m2sqp\" (UID: \"6456a19b-fcb4-4955-aee2-ad30ccc7f4ca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp"
Sep 12 00:50:48.697439 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.697329    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6456a19b-fcb4-4955-aee2-ad30ccc7f4ca-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-m2sqp\" (UID: \"6456a19b-fcb4-4955-aee2-ad30ccc7f4ca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp"
Sep 12 00:50:48.697917 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.697894    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6456a19b-fcb4-4955-aee2-ad30ccc7f4ca-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-m2sqp\" (UID: \"6456a19b-fcb4-4955-aee2-ad30ccc7f4ca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp"
Sep 12 00:50:48.699779 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.699761    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6456a19b-fcb4-4955-aee2-ad30ccc7f4ca-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-m2sqp\" (UID: \"6456a19b-fcb4-4955-aee2-ad30ccc7f4ca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp"
Sep 12 00:50:48.710413 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.710393    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpgz8\" (UniqueName: \"kubernetes.io/projected/6456a19b-fcb4-4955-aee2-ad30ccc7f4ca-kube-api-access-xpgz8\") pod \"kuadrant-console-plugin-fbdd4f445-m2sqp\" (UID: \"6456a19b-fcb4-4955-aee2-ad30ccc7f4ca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp"
Sep 12 00:50:48.833847 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:48.833768    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp"
Sep 12 00:50:49.113579 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:49.113499    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp"]
Sep 12 00:50:49.115755 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:50:49.115729    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6456a19b_fcb4_4955_aee2_ad30ccc7f4ca.slice/crio-67d56a0db39276a56b4343c20ccb93a497c293e2cae9a54a51dfb0db78ea40f3 WatchSource:0}: Error finding container 67d56a0db39276a56b4343c20ccb93a497c293e2cae9a54a51dfb0db78ea40f3: Status 404 returned error can't find the container with id 67d56a0db39276a56b4343c20ccb93a497c293e2cae9a54a51dfb0db78ea40f3
Sep 12 00:50:49.133382 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:49.133356    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp" event={"ID":"6456a19b-fcb4-4955-aee2-ad30ccc7f4ca","Type":"ContainerStarted","Data":"67d56a0db39276a56b4343c20ccb93a497c293e2cae9a54a51dfb0db78ea40f3"}
Sep 12 00:50:49.770574 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:49.770539    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:49.870640 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:49.870604    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:54.158817 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:54.158776    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp" event={"ID":"6456a19b-fcb4-4955-aee2-ad30ccc7f4ca","Type":"ContainerStarted","Data":"1e2d0e5122ec58ab8d56df7ab6d51cf799edac1cbe590ca1d0279b146d780c9f"}
Sep 12 00:50:54.470555 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:54.470463    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:54.568529 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:54.568467    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:54.668073 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:50:54.668035    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:02.373109 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:02.373081    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:02.472867 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:02.472826    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:02.767974 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:02.767917    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:02.868789 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:02.868757    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:03.543723 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:03.543682    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:03.644521 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:03.644491    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:05.165262 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:05.165224    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:05.230773 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:05.230742    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:09.363847 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:09.363816    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:09.468249 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:09.468217    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:14.219181 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:14.219147    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:14.314850 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:14.314817    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:14.417720 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:14.417665    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:27.714253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:27.714219    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:28.161215 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:28.161183    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:43.751230 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.751167    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m2sqp" podStartSLOduration=51.242790827 podStartE2EDuration="55.751146633s" totalImagesPullingTime="4.508355806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:50:54.175063206 +0000 UTC m=+458.735514885" watchObservedRunningTime="2026-09-12 00:51:43.751146633 +0000 UTC m=+508.311598302"
Sep 12 00:51:43.752523 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.752503    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pf794"]
Sep 12 00:51:43.755737 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.755719    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-pf794"
Sep 12 00:51:43.757771 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.757745    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:43.758178 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.758162    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 12 00:51:43.766101 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.766074    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pf794"]
Sep 12 00:51:43.848049 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.848017    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pf794"]
Sep 12 00:51:43.869246 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.869215    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:43.888256 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.888227    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2e3074aa-709a-4e36-a0aa-b6393818404e-config-file\") pod \"limitador-limitador-8588749977-pf794\" (UID: \"2e3074aa-709a-4e36-a0aa-b6393818404e\") " pod="kuadrant-system/limitador-limitador-8588749977-pf794"
Sep 12 00:51:43.888256 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.888260    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6btkg\" (UniqueName: \"kubernetes.io/projected/2e3074aa-709a-4e36-a0aa-b6393818404e-kube-api-access-6btkg\") pod \"limitador-limitador-8588749977-pf794\" (UID: \"2e3074aa-709a-4e36-a0aa-b6393818404e\") " pod="kuadrant-system/limitador-limitador-8588749977-pf794"
Sep 12 00:51:43.969280 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.969219    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:43.989339 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.989312    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2e3074aa-709a-4e36-a0aa-b6393818404e-config-file\") pod \"limitador-limitador-8588749977-pf794\" (UID: \"2e3074aa-709a-4e36-a0aa-b6393818404e\") " pod="kuadrant-system/limitador-limitador-8588749977-pf794"
Sep 12 00:51:43.989482 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.989345    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6btkg\" (UniqueName: \"kubernetes.io/projected/2e3074aa-709a-4e36-a0aa-b6393818404e-kube-api-access-6btkg\") pod \"limitador-limitador-8588749977-pf794\" (UID: \"2e3074aa-709a-4e36-a0aa-b6393818404e\") " pod="kuadrant-system/limitador-limitador-8588749977-pf794"
Sep 12 00:51:43.989902 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.989884    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2e3074aa-709a-4e36-a0aa-b6393818404e-config-file\") pod \"limitador-limitador-8588749977-pf794\" (UID: \"2e3074aa-709a-4e36-a0aa-b6393818404e\") " pod="kuadrant-system/limitador-limitador-8588749977-pf794"
Sep 12 00:51:43.998049 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:43.998018    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6btkg\" (UniqueName: \"kubernetes.io/projected/2e3074aa-709a-4e36-a0aa-b6393818404e-kube-api-access-6btkg\") pod \"limitador-limitador-8588749977-pf794\" (UID: \"2e3074aa-709a-4e36-a0aa-b6393818404e\") " pod="kuadrant-system/limitador-limitador-8588749977-pf794"
Sep 12 00:51:44.066075 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:44.065989    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-pf794"
Sep 12 00:51:44.338040 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:44.338012    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pf794"]
Sep 12 00:51:44.340569 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:51:44.340541    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3074aa_709a_4e36_a0aa_b6393818404e.slice/crio-3b0084999a06a87cab752e2cab6c47e5a2df92738e5e7dd064a77d86103b74a2 WatchSource:0}: Error finding container 3b0084999a06a87cab752e2cab6c47e5a2df92738e5e7dd064a77d86103b74a2: Status 404 returned error can't find the container with id 3b0084999a06a87cab752e2cab6c47e5a2df92738e5e7dd064a77d86103b74a2
Sep 12 00:51:44.377064 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:44.377028    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-pf794" event={"ID":"2e3074aa-709a-4e36-a0aa-b6393818404e","Type":"ContainerStarted","Data":"3b0084999a06a87cab752e2cab6c47e5a2df92738e5e7dd064a77d86103b74a2"}
Sep 12 00:51:44.600911 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:44.600835    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:44.682941 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:44.682909    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:44.683719 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:44.683701    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-d5zvv"]
Sep 12 00:51:44.687930 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:44.687911    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d5zvv"
Sep 12 00:51:44.690221 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:44.690205    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dh8xs\""
Sep 12 00:51:44.693802 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:44.693781    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d5zvv"]
Sep 12 00:51:44.795572 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:44.795541    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4phlg\" (UniqueName: \"kubernetes.io/projected/9ada202f-4840-43ab-854c-1def48287251-kube-api-access-4phlg\") pod \"authorino-59c974bd48-d5zvv\" (UID: \"9ada202f-4840-43ab-854c-1def48287251\") " pod="kuadrant-system/authorino-59c974bd48-d5zvv"
Sep 12 00:51:44.896089 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:44.896011    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4phlg\" (UniqueName: \"kubernetes.io/projected/9ada202f-4840-43ab-854c-1def48287251-kube-api-access-4phlg\") pod \"authorino-59c974bd48-d5zvv\" (UID: \"9ada202f-4840-43ab-854c-1def48287251\") " pod="kuadrant-system/authorino-59c974bd48-d5zvv"
Sep 12 00:51:44.905673 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:44.905619    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4phlg\" (UniqueName: \"kubernetes.io/projected/9ada202f-4840-43ab-854c-1def48287251-kube-api-access-4phlg\") pod \"authorino-59c974bd48-d5zvv\" (UID: \"9ada202f-4840-43ab-854c-1def48287251\") " pod="kuadrant-system/authorino-59c974bd48-d5zvv"
Sep 12 00:51:44.997581 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:44.997547    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d5zvv"
Sep 12 00:51:45.282126 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:45.282097    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d5zvv"]
Sep 12 00:51:45.285034 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:51:45.285004    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ada202f_4840_43ab_854c_1def48287251.slice/crio-878c2ed6398b8d2fd71cc9bf095db3dd58b35da57cbbef9c4fbc2a2b17174dc5 WatchSource:0}: Error finding container 878c2ed6398b8d2fd71cc9bf095db3dd58b35da57cbbef9c4fbc2a2b17174dc5: Status 404 returned error can't find the container with id 878c2ed6398b8d2fd71cc9bf095db3dd58b35da57cbbef9c4fbc2a2b17174dc5
Sep 12 00:51:45.384146 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:45.384113    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d5zvv" event={"ID":"9ada202f-4840-43ab-854c-1def48287251","Type":"ContainerStarted","Data":"878c2ed6398b8d2fd71cc9bf095db3dd58b35da57cbbef9c4fbc2a2b17174dc5"}
Sep 12 00:51:47.394299 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:47.394250    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-pf794" event={"ID":"2e3074aa-709a-4e36-a0aa-b6393818404e","Type":"ContainerStarted","Data":"698a6643dfce6e0a8ac77754e16cf52f47711b52e3e4c11507b482a2cab89b42"}
Sep 12 00:51:47.394749 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:47.394426    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-pf794"
Sep 12 00:51:47.411884 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:47.411766    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-pf794" podStartSLOduration=2.328941496 podStartE2EDuration="4.411744712s" totalImagesPullingTime="2.082803216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:51:47.410370545 +0000 UTC m=+511.970822233" watchObservedRunningTime="2026-09-12 00:51:47.411744712 +0000 UTC m=+511.972196381"
Sep 12 00:51:48.400284 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:48.400248    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d5zvv" event={"ID":"9ada202f-4840-43ab-854c-1def48287251","Type":"ContainerStarted","Data":"cbc9fccb43f46efd0fbf0dbbc49c8d29b824bc9d7d1fdac2e792896cc1b77deb"}
Sep 12 00:51:48.419040 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:48.418993    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-d5zvv" podStartSLOduration=2.050470815 podStartE2EDuration="4.418978672s" totalImagesPullingTime="2.368507857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:51:48.417772315 +0000 UTC m=+512.978223984" watchObservedRunningTime="2026-09-12 00:51:48.418978672 +0000 UTC m=+512.979430337"
Sep 12 00:51:57.298702 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:57.298666    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pf794"]
Sep 12 00:51:57.299122 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:57.298887    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-pf794" podUID="2e3074aa-709a-4e36-a0aa-b6393818404e" containerName="limitador" containerID="cri-o://698a6643dfce6e0a8ac77754e16cf52f47711b52e3e4c11507b482a2cab89b42" gracePeriod=30
Sep 12 00:51:57.299658 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:57.299636    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-pf794"
Sep 12 00:51:58.233253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.233228    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-pf794"
Sep 12 00:51:58.305941 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.305905    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2e3074aa-709a-4e36-a0aa-b6393818404e-config-file\") pod \"2e3074aa-709a-4e36-a0aa-b6393818404e\" (UID: \"2e3074aa-709a-4e36-a0aa-b6393818404e\") "
Sep 12 00:51:58.305941 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.305962    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6btkg\" (UniqueName: \"kubernetes.io/projected/2e3074aa-709a-4e36-a0aa-b6393818404e-kube-api-access-6btkg\") pod \"2e3074aa-709a-4e36-a0aa-b6393818404e\" (UID: \"2e3074aa-709a-4e36-a0aa-b6393818404e\") "
Sep 12 00:51:58.306451 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.306249    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3074aa-709a-4e36-a0aa-b6393818404e-config-file" (OuterVolumeSpecName: "config-file") pod "2e3074aa-709a-4e36-a0aa-b6393818404e" (UID: "2e3074aa-709a-4e36-a0aa-b6393818404e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:51:58.308331 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.308308    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3074aa-709a-4e36-a0aa-b6393818404e-kube-api-access-6btkg" (OuterVolumeSpecName: "kube-api-access-6btkg") pod "2e3074aa-709a-4e36-a0aa-b6393818404e" (UID: "2e3074aa-709a-4e36-a0aa-b6393818404e"). InnerVolumeSpecName "kube-api-access-6btkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:51:58.406707 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.406684    2567 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2e3074aa-709a-4e36-a0aa-b6393818404e-config-file\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:51:58.406707 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.406708    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6btkg\" (UniqueName: \"kubernetes.io/projected/2e3074aa-709a-4e36-a0aa-b6393818404e-kube-api-access-6btkg\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:51:58.445293 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.445266    2567 generic.go:356] "Generic (PLEG): container finished" podID="2e3074aa-709a-4e36-a0aa-b6393818404e" containerID="698a6643dfce6e0a8ac77754e16cf52f47711b52e3e4c11507b482a2cab89b42" exitCode=0
Sep 12 00:51:58.445401 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.445317    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-pf794"
Sep 12 00:51:58.445401 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.445349    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-pf794" event={"ID":"2e3074aa-709a-4e36-a0aa-b6393818404e","Type":"ContainerDied","Data":"698a6643dfce6e0a8ac77754e16cf52f47711b52e3e4c11507b482a2cab89b42"}
Sep 12 00:51:58.445526 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.445402    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-pf794" event={"ID":"2e3074aa-709a-4e36-a0aa-b6393818404e","Type":"ContainerDied","Data":"3b0084999a06a87cab752e2cab6c47e5a2df92738e5e7dd064a77d86103b74a2"}
Sep 12 00:51:58.445526 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.445424    2567 scope.go:117] "RemoveContainer" containerID="698a6643dfce6e0a8ac77754e16cf52f47711b52e3e4c11507b482a2cab89b42"
Sep 12 00:51:58.453983 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.453966    2567 scope.go:117] "RemoveContainer" containerID="698a6643dfce6e0a8ac77754e16cf52f47711b52e3e4c11507b482a2cab89b42"
Sep 12 00:51:58.454236 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:51:58.454218    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698a6643dfce6e0a8ac77754e16cf52f47711b52e3e4c11507b482a2cab89b42\": container with ID starting with 698a6643dfce6e0a8ac77754e16cf52f47711b52e3e4c11507b482a2cab89b42 not found: ID does not exist" containerID="698a6643dfce6e0a8ac77754e16cf52f47711b52e3e4c11507b482a2cab89b42"
Sep 12 00:51:58.454298 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.454244    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698a6643dfce6e0a8ac77754e16cf52f47711b52e3e4c11507b482a2cab89b42"} err="failed to get container status \"698a6643dfce6e0a8ac77754e16cf52f47711b52e3e4c11507b482a2cab89b42\": rpc error: code = NotFound desc = could not find container \"698a6643dfce6e0a8ac77754e16cf52f47711b52e3e4c11507b482a2cab89b42\": container with ID starting with 698a6643dfce6e0a8ac77754e16cf52f47711b52e3e4c11507b482a2cab89b42 not found: ID does not exist"
Sep 12 00:51:58.466119 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.466100    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pf794"]
Sep 12 00:51:58.470689 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:58.470671    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pf794"]
Sep 12 00:51:59.972728 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:51:59.972697    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e3074aa-709a-4e36-a0aa-b6393818404e" path="/var/lib/kubelet/pods/2e3074aa-709a-4e36-a0aa-b6393818404e/volumes"
Sep 12 00:52:01.049938 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.049910    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:01.143795 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.143760    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:01.508000 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.507968    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-t758b"]
Sep 12 00:52:01.508318 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.508304    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e3074aa-709a-4e36-a0aa-b6393818404e" containerName="limitador"
Sep 12 00:52:01.508371 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.508319    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e3074aa-709a-4e36-a0aa-b6393818404e" containerName="limitador"
Sep 12 00:52:01.508413 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.508385    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e3074aa-709a-4e36-a0aa-b6393818404e" containerName="limitador"
Sep 12 00:52:01.512647 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.512629    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t758b"
Sep 12 00:52:01.515445 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.515420    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 12 00:52:01.515559 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.515486    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 12 00:52:01.516389 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.516370    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nrk94\""
Sep 12 00:52:01.516389 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.516386    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:52:01.520135 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.520112    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-t758b"]
Sep 12 00:52:01.532776 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.532754    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/56c87398-ea9e-413d-9d66-e314c3d0e092-data\") pod \"postgres-69fb4594cf-t758b\" (UID: \"56c87398-ea9e-413d-9d66-e314c3d0e092\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t758b"
Sep 12 00:52:01.532860 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.532784    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc92r\" (UniqueName: \"kubernetes.io/projected/56c87398-ea9e-413d-9d66-e314c3d0e092-kube-api-access-qc92r\") pod \"postgres-69fb4594cf-t758b\" (UID: \"56c87398-ea9e-413d-9d66-e314c3d0e092\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t758b"
Sep 12 00:52:01.539539 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.539521    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:01.633456 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.633429    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/56c87398-ea9e-413d-9d66-e314c3d0e092-data\") pod \"postgres-69fb4594cf-t758b\" (UID: \"56c87398-ea9e-413d-9d66-e314c3d0e092\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t758b"
Sep 12 00:52:01.633565 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.633476    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc92r\" (UniqueName: \"kubernetes.io/projected/56c87398-ea9e-413d-9d66-e314c3d0e092-kube-api-access-qc92r\") pod \"postgres-69fb4594cf-t758b\" (UID: \"56c87398-ea9e-413d-9d66-e314c3d0e092\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t758b"
Sep 12 00:52:01.633770 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.633750    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/56c87398-ea9e-413d-9d66-e314c3d0e092-data\") pod \"postgres-69fb4594cf-t758b\" (UID: \"56c87398-ea9e-413d-9d66-e314c3d0e092\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t758b"
Sep 12 00:52:01.642752 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.642729    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc92r\" (UniqueName: \"kubernetes.io/projected/56c87398-ea9e-413d-9d66-e314c3d0e092-kube-api-access-qc92r\") pod \"postgres-69fb4594cf-t758b\" (UID: \"56c87398-ea9e-413d-9d66-e314c3d0e092\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t758b"
Sep 12 00:52:01.823648 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:01.823547    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t758b"
Sep 12 00:52:02.093936 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:02.093913    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-t758b"]
Sep 12 00:52:02.096108 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:52:02.096082    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c87398_ea9e_413d_9d66_e314c3d0e092.slice/crio-9659907c98f78c3ef2fb079d4c9e117b0aa34d63c762633b7deb401eec7d9c27 WatchSource:0}: Error finding container 9659907c98f78c3ef2fb079d4c9e117b0aa34d63c762633b7deb401eec7d9c27: Status 404 returned error can't find the container with id 9659907c98f78c3ef2fb079d4c9e117b0aa34d63c762633b7deb401eec7d9c27
Sep 12 00:52:02.464329 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:02.464297    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t758b" event={"ID":"56c87398-ea9e-413d-9d66-e314c3d0e092","Type":"ContainerStarted","Data":"9659907c98f78c3ef2fb079d4c9e117b0aa34d63c762633b7deb401eec7d9c27"}
Sep 12 00:52:05.482135 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:05.482097    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t758b" event={"ID":"56c87398-ea9e-413d-9d66-e314c3d0e092","Type":"ContainerStarted","Data":"93206c61f87f073d672a1b6e253086da9389b8fbdb8d0e09d613bb1da33e22e9"}
Sep 12 00:52:05.482527 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:05.482253    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t758b"
Sep 12 00:52:05.502214 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:05.502157    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t758b" podStartSLOduration=2.1153089 podStartE2EDuration="4.502136642s" totalImagesPullingTime="2.386827742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:52:05.498621916 +0000 UTC m=+530.059073605" watchObservedRunningTime="2026-09-12 00:52:05.502136642 +0000 UTC m=+530.062588312"
Sep 12 00:52:11.514434 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:11.514354    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t758b"
Sep 12 00:52:11.839057 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:11.838977    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:11.983631 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:11.983602    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:14.451854 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:14.451818    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d5zvv"]
Sep 12 00:52:14.452347 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:14.452033    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-d5zvv" podUID="9ada202f-4840-43ab-854c-1def48287251" containerName="authorino" containerID="cri-o://cbc9fccb43f46efd0fbf0dbbc49c8d29b824bc9d7d1fdac2e792896cc1b77deb" gracePeriod=30
Sep 12 00:52:14.699103 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:14.699077    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d5zvv"
Sep 12 00:52:14.748979 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:14.748890    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4phlg\" (UniqueName: \"kubernetes.io/projected/9ada202f-4840-43ab-854c-1def48287251-kube-api-access-4phlg\") pod \"9ada202f-4840-43ab-854c-1def48287251\" (UID: \"9ada202f-4840-43ab-854c-1def48287251\") "
Sep 12 00:52:14.751005 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:14.750978    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ada202f-4840-43ab-854c-1def48287251-kube-api-access-4phlg" (OuterVolumeSpecName: "kube-api-access-4phlg") pod "9ada202f-4840-43ab-854c-1def48287251" (UID: "9ada202f-4840-43ab-854c-1def48287251"). InnerVolumeSpecName "kube-api-access-4phlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:52:14.849661 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:14.849625    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4phlg\" (UniqueName: \"kubernetes.io/projected/9ada202f-4840-43ab-854c-1def48287251-kube-api-access-4phlg\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:52:15.528592 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:15.528561    2567 generic.go:356] "Generic (PLEG): container finished" podID="9ada202f-4840-43ab-854c-1def48287251" containerID="cbc9fccb43f46efd0fbf0dbbc49c8d29b824bc9d7d1fdac2e792896cc1b77deb" exitCode=0
Sep 12 00:52:15.529054 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:15.528600    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d5zvv"
Sep 12 00:52:15.529054 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:15.528611    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d5zvv" event={"ID":"9ada202f-4840-43ab-854c-1def48287251","Type":"ContainerDied","Data":"cbc9fccb43f46efd0fbf0dbbc49c8d29b824bc9d7d1fdac2e792896cc1b77deb"}
Sep 12 00:52:15.529054 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:15.528651    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d5zvv" event={"ID":"9ada202f-4840-43ab-854c-1def48287251","Type":"ContainerDied","Data":"878c2ed6398b8d2fd71cc9bf095db3dd58b35da57cbbef9c4fbc2a2b17174dc5"}
Sep 12 00:52:15.529054 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:15.528673    2567 scope.go:117] "RemoveContainer" containerID="cbc9fccb43f46efd0fbf0dbbc49c8d29b824bc9d7d1fdac2e792896cc1b77deb"
Sep 12 00:52:15.536767 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:15.536752    2567 scope.go:117] "RemoveContainer" containerID="cbc9fccb43f46efd0fbf0dbbc49c8d29b824bc9d7d1fdac2e792896cc1b77deb"
Sep 12 00:52:15.537016 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:52:15.536997    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc9fccb43f46efd0fbf0dbbc49c8d29b824bc9d7d1fdac2e792896cc1b77deb\": container with ID starting with cbc9fccb43f46efd0fbf0dbbc49c8d29b824bc9d7d1fdac2e792896cc1b77deb not found: ID does not exist" containerID="cbc9fccb43f46efd0fbf0dbbc49c8d29b824bc9d7d1fdac2e792896cc1b77deb"
Sep 12 00:52:15.537079 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:15.537023    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc9fccb43f46efd0fbf0dbbc49c8d29b824bc9d7d1fdac2e792896cc1b77deb"} err="failed to get container status \"cbc9fccb43f46efd0fbf0dbbc49c8d29b824bc9d7d1fdac2e792896cc1b77deb\": rpc error: code = NotFound desc = could not find container \"cbc9fccb43f46efd0fbf0dbbc49c8d29b824bc9d7d1fdac2e792896cc1b77deb\": container with ID starting with cbc9fccb43f46efd0fbf0dbbc49c8d29b824bc9d7d1fdac2e792896cc1b77deb not found: ID does not exist"
Sep 12 00:52:15.550785 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:15.550762    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d5zvv"]
Sep 12 00:52:15.556492 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:15.556470    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d5zvv"]
Sep 12 00:52:15.972468 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:15.972432    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ada202f-4840-43ab-854c-1def48287251" path="/var/lib/kubelet/pods/9ada202f-4840-43ab-854c-1def48287251/volumes"
Sep 12 00:52:19.342515 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.342476    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:19.405525 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.405487    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-85c777d7b7-nnzt7"]
Sep 12 00:52:19.405899 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.405881    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ada202f-4840-43ab-854c-1def48287251" containerName="authorino"
Sep 12 00:52:19.406010 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.405901    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ada202f-4840-43ab-854c-1def48287251" containerName="authorino"
Sep 12 00:52:19.406076 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.406022    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ada202f-4840-43ab-854c-1def48287251" containerName="authorino"
Sep 12 00:52:19.411540 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.411519    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:19.415452 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.415429    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 12 00:52:19.416407 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.416384    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-484sv\""
Sep 12 00:52:19.416525 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.416389    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 12 00:52:19.416525 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.416391    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 12 00:52:19.421423 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.421395    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85c777d7b7-nnzt7"]
Sep 12 00:52:19.430033 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.430007    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:19.488720 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.488687    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-metrics-certs\") pod \"maas-controller-85c777d7b7-nnzt7\" (UID: \"2ae15776-977b-4cfd-bf39-366b46eba3d5\") " pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:19.488875 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.488724    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dctl8\" (UniqueName: \"kubernetes.io/projected/2ae15776-977b-4cfd-bf39-366b46eba3d5-kube-api-access-dctl8\") pod \"maas-controller-85c777d7b7-nnzt7\" (UID: \"2ae15776-977b-4cfd-bf39-366b46eba3d5\") " pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:19.488875 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.488825    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-webhook-cert\") pod \"maas-controller-85c777d7b7-nnzt7\" (UID: \"2ae15776-977b-4cfd-bf39-366b46eba3d5\") " pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:19.530625 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.530593    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:19.589708 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.589682    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-webhook-cert\") pod \"maas-controller-85c777d7b7-nnzt7\" (UID: \"2ae15776-977b-4cfd-bf39-366b46eba3d5\") " pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:19.589862 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.589744    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-metrics-certs\") pod \"maas-controller-85c777d7b7-nnzt7\" (UID: \"2ae15776-977b-4cfd-bf39-366b46eba3d5\") " pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:19.589862 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.589770    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dctl8\" (UniqueName: \"kubernetes.io/projected/2ae15776-977b-4cfd-bf39-366b46eba3d5-kube-api-access-dctl8\") pod \"maas-controller-85c777d7b7-nnzt7\" (UID: \"2ae15776-977b-4cfd-bf39-366b46eba3d5\") " pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:19.589862 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:52:19.589838    2567 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 12 00:52:19.589998 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:52:19.589924    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-webhook-cert podName:2ae15776-977b-4cfd-bf39-366b46eba3d5 nodeName:}" failed. No retries permitted until 2026-09-12 00:52:20.089906602 +0000 UTC m=+544.650358251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-webhook-cert") pod "maas-controller-85c777d7b7-nnzt7" (UID: "2ae15776-977b-4cfd-bf39-366b46eba3d5") : secret "maas-controller-webhook-cert" not found
Sep 12 00:52:19.592486 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.592460    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-metrics-certs\") pod \"maas-controller-85c777d7b7-nnzt7\" (UID: \"2ae15776-977b-4cfd-bf39-366b46eba3d5\") " pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:19.602373 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:19.602317    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dctl8\" (UniqueName: \"kubernetes.io/projected/2ae15776-977b-4cfd-bf39-366b46eba3d5-kube-api-access-dctl8\") pod \"maas-controller-85c777d7b7-nnzt7\" (UID: \"2ae15776-977b-4cfd-bf39-366b46eba3d5\") " pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:20.094861 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:20.094825    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-webhook-cert\") pod \"maas-controller-85c777d7b7-nnzt7\" (UID: \"2ae15776-977b-4cfd-bf39-366b46eba3d5\") " pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:20.097379 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:20.097351    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-webhook-cert\") pod \"maas-controller-85c777d7b7-nnzt7\" (UID: \"2ae15776-977b-4cfd-bf39-366b46eba3d5\") " pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:20.152057 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:20.152030    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:20.322670 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:20.322634    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:20.592490 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:20.592414    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85c777d7b7-nnzt7"]
Sep 12 00:52:20.594796 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:52:20.594772    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae15776_977b_4cfd_bf39_366b46eba3d5.slice/crio-e57c094ca6e44b65997c8cb7b095d2b23b1afb29c187a6ec1bba9d05c30ccd42 WatchSource:0}: Error finding container e57c094ca6e44b65997c8cb7b095d2b23b1afb29c187a6ec1bba9d05c30ccd42: Status 404 returned error can't find the container with id e57c094ca6e44b65997c8cb7b095d2b23b1afb29c187a6ec1bba9d05c30ccd42
Sep 12 00:52:21.556929 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:21.556888    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85c777d7b7-nnzt7" event={"ID":"2ae15776-977b-4cfd-bf39-366b46eba3d5","Type":"ContainerStarted","Data":"e57c094ca6e44b65997c8cb7b095d2b23b1afb29c187a6ec1bba9d05c30ccd42"}
Sep 12 00:52:23.209993 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:23.209963    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:23.307515 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:23.307435    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:23.566543 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:23.566451    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85c777d7b7-nnzt7" event={"ID":"2ae15776-977b-4cfd-bf39-366b46eba3d5","Type":"ContainerStarted","Data":"30529c3f1250ea7a48c9eea34733bfb3a29c90c7b2aad6e85286c0866845bd8b"}
Sep 12 00:52:23.566543 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:23.566525    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:23.592904 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:23.592852    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-85c777d7b7-nnzt7" podStartSLOduration=2.153584453 podStartE2EDuration="4.592836194s" totalImagesPullingTime="2.439251741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:52:23.585525025 +0000 UTC m=+548.145976743" watchObservedRunningTime="2026-09-12 00:52:23.592836194 +0000 UTC m=+548.153287862"
Sep 12 00:52:34.573894 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:34.573863    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:34.730137 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:34.730091    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85c777d7b7-nnzt7"]
Sep 12 00:52:34.730341 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:34.730316    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-85c777d7b7-nnzt7" podUID="2ae15776-977b-4cfd-bf39-366b46eba3d5" containerName="manager" containerID="cri-o://30529c3f1250ea7a48c9eea34733bfb3a29c90c7b2aad6e85286c0866845bd8b" gracePeriod=30
Sep 12 00:52:34.970805 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:34.970783    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:35.120681 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.120590    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-metrics-certs\") pod \"2ae15776-977b-4cfd-bf39-366b46eba3d5\" (UID: \"2ae15776-977b-4cfd-bf39-366b46eba3d5\") "
Sep 12 00:52:35.120855 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.120689    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dctl8\" (UniqueName: \"kubernetes.io/projected/2ae15776-977b-4cfd-bf39-366b46eba3d5-kube-api-access-dctl8\") pod \"2ae15776-977b-4cfd-bf39-366b46eba3d5\" (UID: \"2ae15776-977b-4cfd-bf39-366b46eba3d5\") "
Sep 12 00:52:35.120855 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.120740    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-webhook-cert\") pod \"2ae15776-977b-4cfd-bf39-366b46eba3d5\" (UID: \"2ae15776-977b-4cfd-bf39-366b46eba3d5\") "
Sep 12 00:52:35.123085 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.123059    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2ae15776-977b-4cfd-bf39-366b46eba3d5" (UID: "2ae15776-977b-4cfd-bf39-366b46eba3d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:52:35.123371 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.123347    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae15776-977b-4cfd-bf39-366b46eba3d5-kube-api-access-dctl8" (OuterVolumeSpecName: "kube-api-access-dctl8") pod "2ae15776-977b-4cfd-bf39-366b46eba3d5" (UID: "2ae15776-977b-4cfd-bf39-366b46eba3d5"). InnerVolumeSpecName "kube-api-access-dctl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:52:35.123468 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.123372    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2ae15776-977b-4cfd-bf39-366b46eba3d5" (UID: "2ae15776-977b-4cfd-bf39-366b46eba3d5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:52:35.222261 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.222211    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dctl8\" (UniqueName: \"kubernetes.io/projected/2ae15776-977b-4cfd-bf39-366b46eba3d5-kube-api-access-dctl8\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:52:35.222261 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.222256    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-webhook-cert\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:52:35.222261 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.222266    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae15776-977b-4cfd-bf39-366b46eba3d5-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:52:35.619623 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.619590    2567 generic.go:356] "Generic (PLEG): container finished" podID="2ae15776-977b-4cfd-bf39-366b46eba3d5" containerID="30529c3f1250ea7a48c9eea34733bfb3a29c90c7b2aad6e85286c0866845bd8b" exitCode=0
Sep 12 00:52:35.620104 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.619639    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85c777d7b7-nnzt7"
Sep 12 00:52:35.620104 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.619662    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85c777d7b7-nnzt7" event={"ID":"2ae15776-977b-4cfd-bf39-366b46eba3d5","Type":"ContainerDied","Data":"30529c3f1250ea7a48c9eea34733bfb3a29c90c7b2aad6e85286c0866845bd8b"}
Sep 12 00:52:35.620104 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.619703    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85c777d7b7-nnzt7" event={"ID":"2ae15776-977b-4cfd-bf39-366b46eba3d5","Type":"ContainerDied","Data":"e57c094ca6e44b65997c8cb7b095d2b23b1afb29c187a6ec1bba9d05c30ccd42"}
Sep 12 00:52:35.620104 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.619724    2567 scope.go:117] "RemoveContainer" containerID="30529c3f1250ea7a48c9eea34733bfb3a29c90c7b2aad6e85286c0866845bd8b"
Sep 12 00:52:35.629027 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.629006    2567 scope.go:117] "RemoveContainer" containerID="30529c3f1250ea7a48c9eea34733bfb3a29c90c7b2aad6e85286c0866845bd8b"
Sep 12 00:52:35.629304 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:52:35.629287    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30529c3f1250ea7a48c9eea34733bfb3a29c90c7b2aad6e85286c0866845bd8b\": container with ID starting with 30529c3f1250ea7a48c9eea34733bfb3a29c90c7b2aad6e85286c0866845bd8b not found: ID does not exist" containerID="30529c3f1250ea7a48c9eea34733bfb3a29c90c7b2aad6e85286c0866845bd8b"
Sep 12 00:52:35.629366 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.629312    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30529c3f1250ea7a48c9eea34733bfb3a29c90c7b2aad6e85286c0866845bd8b"} err="failed to get container status \"30529c3f1250ea7a48c9eea34733bfb3a29c90c7b2aad6e85286c0866845bd8b\": rpc error: code = NotFound desc = could not find container \"30529c3f1250ea7a48c9eea34733bfb3a29c90c7b2aad6e85286c0866845bd8b\": container with ID starting with 30529c3f1250ea7a48c9eea34733bfb3a29c90c7b2aad6e85286c0866845bd8b not found: ID does not exist"
Sep 12 00:52:35.639747 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.639721    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85c777d7b7-nnzt7"]
Sep 12 00:52:35.643676 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.643657    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-85c777d7b7-nnzt7"]
Sep 12 00:52:35.978283 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:35.978255    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ae15776-977b-4cfd-bf39-366b46eba3d5" path="/var/lib/kubelet/pods/2ae15776-977b-4cfd-bf39-366b46eba3d5/volumes"
Sep 12 00:52:55.805125 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:55.805050    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:55.902234 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:55.902201    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:57.210550 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.210519    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:57.306849 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.306805    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:57.412207 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.412176    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:57.490891 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.490808    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:57.494671 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.494643    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"]
Sep 12 00:52:57.495056 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.495039    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae15776-977b-4cfd-bf39-366b46eba3d5" containerName="manager"
Sep 12 00:52:57.495143 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.495078    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae15776-977b-4cfd-bf39-366b46eba3d5" containerName="manager"
Sep 12 00:52:57.495203 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.495187    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae15776-977b-4cfd-bf39-366b46eba3d5" containerName="manager"
Sep 12 00:52:57.501367 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.501348    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:52:57.503859 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.503831    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483879f7-8946-4dc9-950f-64f8082d9651-maas-api-tls\") pod \"maas-api-6cc46ff6dc-bwqhl\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:52:57.503999 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.503876    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483879f7-8946-4dc9-950f-64f8082d9651-metrics-certs\") pod \"maas-api-6cc46ff6dc-bwqhl\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:52:57.503999 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.503895    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6sll\" (UniqueName: \"kubernetes.io/projected/483879f7-8946-4dc9-950f-64f8082d9651-kube-api-access-h6sll\") pod \"maas-api-6cc46ff6dc-bwqhl\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:52:57.503999 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.503937    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483879f7-8946-4dc9-950f-64f8082d9651-openshift-service-ca\") pod \"maas-api-6cc46ff6dc-bwqhl\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:52:57.505039 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.505016    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k2gll\""
Sep 12 00:52:57.505152 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.505061    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 12 00:52:57.505152 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.505020    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 12 00:52:57.509729 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.509710    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"]
Sep 12 00:52:57.605110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.605074    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483879f7-8946-4dc9-950f-64f8082d9651-maas-api-tls\") pod \"maas-api-6cc46ff6dc-bwqhl\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:52:57.605110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.605111    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483879f7-8946-4dc9-950f-64f8082d9651-metrics-certs\") pod \"maas-api-6cc46ff6dc-bwqhl\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:52:57.605324 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.605128    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6sll\" (UniqueName: \"kubernetes.io/projected/483879f7-8946-4dc9-950f-64f8082d9651-kube-api-access-h6sll\") pod \"maas-api-6cc46ff6dc-bwqhl\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:52:57.605324 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.605171    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483879f7-8946-4dc9-950f-64f8082d9651-openshift-service-ca\") pod \"maas-api-6cc46ff6dc-bwqhl\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:52:57.605988 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.605909    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483879f7-8946-4dc9-950f-64f8082d9651-openshift-service-ca\") pod \"maas-api-6cc46ff6dc-bwqhl\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:52:57.607937 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.607911    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483879f7-8946-4dc9-950f-64f8082d9651-maas-api-tls\") pod \"maas-api-6cc46ff6dc-bwqhl\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:52:57.608096 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.608001    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483879f7-8946-4dc9-950f-64f8082d9651-metrics-certs\") pod \"maas-api-6cc46ff6dc-bwqhl\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:52:57.613280 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.613255    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6sll\" (UniqueName: \"kubernetes.io/projected/483879f7-8946-4dc9-950f-64f8082d9651-kube-api-access-h6sll\") pod \"maas-api-6cc46ff6dc-bwqhl\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:52:57.777478 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.777395    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:57.813047 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.813011    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:52:57.878372 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.878258    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:57.952992 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.952931    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5"]
Sep 12 00:52:57.957725 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.957704    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5"
Sep 12 00:52:57.960342 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.960319    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 12 00:52:57.960461 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.960400    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hvkvx\""
Sep 12 00:52:57.977048 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:57.977005    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5"]
Sep 12 00:52:58.008867 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:58.008837    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4579cc0d-433e-486c-a553-d0c76bda8b0d-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-rzbx5\" (UID: \"4579cc0d-433e-486c-a553-d0c76bda8b0d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5"
Sep 12 00:52:58.009037 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:58.008901    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d82qg\" (UniqueName: \"kubernetes.io/projected/4579cc0d-433e-486c-a553-d0c76bda8b0d-kube-api-access-d82qg\") pod \"payload-pre-processing-59fbc47bf8-rzbx5\" (UID: \"4579cc0d-433e-486c-a553-d0c76bda8b0d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5"
Sep 12 00:52:58.085671 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:58.085647    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"]
Sep 12 00:52:58.087700 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:52:58.087672    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483879f7_8946_4dc9_950f_64f8082d9651.slice/crio-f7b7a3d02b8cfd3c1132da246940f3b7f710a9c70e683e18fe779214b1b0aa25 WatchSource:0}: Error finding container f7b7a3d02b8cfd3c1132da246940f3b7f710a9c70e683e18fe779214b1b0aa25: Status 404 returned error can't find the container with id f7b7a3d02b8cfd3c1132da246940f3b7f710a9c70e683e18fe779214b1b0aa25
Sep 12 00:52:58.109420 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:58.109399    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d82qg\" (UniqueName: \"kubernetes.io/projected/4579cc0d-433e-486c-a553-d0c76bda8b0d-kube-api-access-d82qg\") pod \"payload-pre-processing-59fbc47bf8-rzbx5\" (UID: \"4579cc0d-433e-486c-a553-d0c76bda8b0d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5"
Sep 12 00:52:58.109537 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:58.109453    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4579cc0d-433e-486c-a553-d0c76bda8b0d-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-rzbx5\" (UID: \"4579cc0d-433e-486c-a553-d0c76bda8b0d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5"
Sep 12 00:52:58.110074 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:58.110056    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4579cc0d-433e-486c-a553-d0c76bda8b0d-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-rzbx5\" (UID: \"4579cc0d-433e-486c-a553-d0c76bda8b0d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5"
Sep 12 00:52:58.119630 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:58.119608    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d82qg\" (UniqueName: \"kubernetes.io/projected/4579cc0d-433e-486c-a553-d0c76bda8b0d-kube-api-access-d82qg\") pod \"payload-pre-processing-59fbc47bf8-rzbx5\" (UID: \"4579cc0d-433e-486c-a553-d0c76bda8b0d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5"
Sep 12 00:52:58.275471 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:58.275442    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5"
Sep 12 00:52:58.546302 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:58.546250    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5"]
Sep 12 00:52:58.548298 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:52:58.548267    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4579cc0d_433e_486c_a553_d0c76bda8b0d.slice/crio-168c500f116a0556557c4a428b062abdde1068cdcc5501d65896f62a2534f2a3 WatchSource:0}: Error finding container 168c500f116a0556557c4a428b062abdde1068cdcc5501d65896f62a2534f2a3: Status 404 returned error can't find the container with id 168c500f116a0556557c4a428b062abdde1068cdcc5501d65896f62a2534f2a3
Sep 12 00:52:58.728925 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:58.728885    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl" event={"ID":"483879f7-8946-4dc9-950f-64f8082d9651","Type":"ContainerStarted","Data":"f7b7a3d02b8cfd3c1132da246940f3b7f710a9c70e683e18fe779214b1b0aa25"}
Sep 12 00:52:58.730088 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:52:58.730063    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5" event={"ID":"4579cc0d-433e-486c-a553-d0c76bda8b0d","Type":"ContainerStarted","Data":"168c500f116a0556557c4a428b062abdde1068cdcc5501d65896f62a2534f2a3"}
Sep 12 00:53:01.746753 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:01.746675    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl" event={"ID":"483879f7-8946-4dc9-950f-64f8082d9651","Type":"ContainerStarted","Data":"e6496b37c642744a553bd3953cdcb0d5c11135d4a87705d4fd2350a2a745b3f4"}
Sep 12 00:53:01.747254 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:01.746870    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:53:01.748357 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:01.748334    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5" event={"ID":"4579cc0d-433e-486c-a553-d0c76bda8b0d","Type":"ContainerStarted","Data":"e2dde03023179d26b56c89e7bfa2f50ebbee55857a93e401151146ceed964d19"}
Sep 12 00:53:01.748458 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:01.748386    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5"
Sep 12 00:53:01.766833 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:01.766785    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl" podStartSLOduration=1.85510267 podStartE2EDuration="4.766772861s" totalImagesPullingTime="2.911670191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:53:01.765143737 +0000 UTC m=+586.325595417" watchObservedRunningTime="2026-09-12 00:53:01.766772861 +0000 UTC m=+586.327224530"
Sep 12 00:53:01.783818 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:01.783771    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5" podStartSLOduration=2.287192223 podStartE2EDuration="4.783757726s" totalImagesPullingTime="2.496565503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:53:01.782861879 +0000 UTC m=+586.343313552" watchObservedRunningTime="2026-09-12 00:53:01.783757726 +0000 UTC m=+586.344209393"
Sep 12 00:53:07.758508 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:07.758472    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 00:53:08.804885 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:08.804846    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:08.897242 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:08.897209    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:12.755776 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:12.755741    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rzbx5"
Sep 12 00:53:15.870002 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:15.869975    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 00:53:15.870564 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:15.870546    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 00:53:17.151258 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:17.151224    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:17.252278 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:17.252246    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:20.679470 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:20.679435    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:26.501233 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:26.501202    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:43.134492 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.134405    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"]
Sep 12 00:53:43.137872 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.137845    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.140514 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.140491    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 12 00:53:43.140635 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.140556    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 12 00:53:43.140784 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.140770    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 12 00:53:43.141902 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.141884    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6nb8d\""
Sep 12 00:53:43.151366 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.151343    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"]
Sep 12 00:53:43.171142 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.171104    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:43.206688 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.206657    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab49529a-5f51-43a6-9314-372c9c54581e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.206688 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.206692    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab49529a-5f51-43a6-9314-372c9c54581e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.206876 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.206722    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab49529a-5f51-43a6-9314-372c9c54581e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.206876 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.206737    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab49529a-5f51-43a6-9314-372c9c54581e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.206876 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.206793    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hzbm\" (UniqueName: \"kubernetes.io/projected/ab49529a-5f51-43a6-9314-372c9c54581e-kube-api-access-7hzbm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.206876 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.206842    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab49529a-5f51-43a6-9314-372c9c54581e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.308073 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.308033    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hzbm\" (UniqueName: \"kubernetes.io/projected/ab49529a-5f51-43a6-9314-372c9c54581e-kube-api-access-7hzbm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.308073 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.308079    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab49529a-5f51-43a6-9314-372c9c54581e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.308290 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.308153    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab49529a-5f51-43a6-9314-372c9c54581e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.308290 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.308179    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab49529a-5f51-43a6-9314-372c9c54581e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.308290 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.308224    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab49529a-5f51-43a6-9314-372c9c54581e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.308290 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.308249    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab49529a-5f51-43a6-9314-372c9c54581e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.308572 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.308547    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab49529a-5f51-43a6-9314-372c9c54581e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.308639 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.308595    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab49529a-5f51-43a6-9314-372c9c54581e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.308682 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.308646    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab49529a-5f51-43a6-9314-372c9c54581e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.310498 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.310476    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab49529a-5f51-43a6-9314-372c9c54581e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.310670 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.310655    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab49529a-5f51-43a6-9314-372c9c54581e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.316935 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.316916    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hzbm\" (UniqueName: \"kubernetes.io/projected/ab49529a-5f51-43a6-9314-372c9c54581e-kube-api-access-7hzbm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-brqqx\" (UID: \"ab49529a-5f51-43a6-9314-372c9c54581e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.448130 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.448092    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:43.722811 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.722785    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"]
Sep 12 00:53:43.725791 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:53:43.725769    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab49529a_5f51_43a6_9314_372c9c54581e.slice/crio-489ad8b225803421c1c68d02af95b61748363d1f66886fa7096e3f64897d856e WatchSource:0}: Error finding container 489ad8b225803421c1c68d02af95b61748363d1f66886fa7096e3f64897d856e: Status 404 returned error can't find the container with id 489ad8b225803421c1c68d02af95b61748363d1f66886fa7096e3f64897d856e
Sep 12 00:53:43.727608 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.727590    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:53:43.939969 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:43.939913    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx" event={"ID":"ab49529a-5f51-43a6-9314-372c9c54581e","Type":"ContainerStarted","Data":"489ad8b225803421c1c68d02af95b61748363d1f66886fa7096e3f64897d856e"}
Sep 12 00:53:44.945351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:44.945313    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx" event={"ID":"ab49529a-5f51-43a6-9314-372c9c54581e","Type":"ContainerStarted","Data":"f360270c09101e73762281e0d7c49280483dcd17fbb4facd9187f5ba474cdfa1"}
Sep 12 00:53:44.964853 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:44.964806    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx" podStartSLOduration=0.960646564 podStartE2EDuration="1.96479273s" totalImagesPullingTime="1.004146166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:53:44.963013002 +0000 UTC m=+629.523464696" watchObservedRunningTime="2026-09-12 00:53:44.96479273 +0000 UTC m=+629.525244397"
Sep 12 00:53:48.258910 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:48.258864    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:53.448506 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:53.448471    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:53.448881 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:53.448621    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:53.461233 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:53.461208    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:54.001719 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:54.001690    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-brqqx"
Sep 12 00:53:55.388740 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:55.388709    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:55.491051 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:53:55.491010    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:54:03.443826 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.443790    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"]
Sep 12 00:54:03.448468 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.448448    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.450935 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.450915    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 12 00:54:03.458107 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.458082    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"]
Sep 12 00:54:03.527198 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.527129    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:54:03.574289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.574256    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.574460 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.574296    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.574460 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.574414    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.574460 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.574441    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bxpz\" (UniqueName: \"kubernetes.io/projected/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-kube-api-access-5bxpz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.574584 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.574472    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.574584 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.574512    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.675621 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.675577    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.675621 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.675624    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bxpz\" (UniqueName: \"kubernetes.io/projected/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-kube-api-access-5bxpz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.675877 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.675656    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.675877 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.675687    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.675877 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.675714    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.676090 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.675862    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.676156 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.676106    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.676156 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.676124    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.676263 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.676176    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.678238 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.678211    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.678609 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.678590    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.684027 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.684007    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bxpz\" (UniqueName: \"kubernetes.io/projected/31a801bd-4fc0-4ea8-ad02-98e22e6c2f79-kube-api-access-5bxpz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhpzd\" (UID: \"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:03.760416 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:03.760320    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:04.034012 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:04.033991    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"]
Sep 12 00:54:04.036033 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:54:04.036008    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a801bd_4fc0_4ea8_ad02_98e22e6c2f79.slice/crio-c8ee6223f7790cd14c67b960c198d5635e386c79eef963927fe5cc401ab81af3 WatchSource:0}: Error finding container c8ee6223f7790cd14c67b960c198d5635e386c79eef963927fe5cc401ab81af3: Status 404 returned error can't find the container with id c8ee6223f7790cd14c67b960c198d5635e386c79eef963927fe5cc401ab81af3
Sep 12 00:54:05.041394 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:05.041351    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd" event={"ID":"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79","Type":"ContainerStarted","Data":"52c52ef291c0b5ba632f93e0869ae617c2fb7dd9f6688e30f59d8b25b05a10c7"}
Sep 12 00:54:05.041769 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:05.041412    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd" event={"ID":"31a801bd-4fc0-4ea8-ad02-98e22e6c2f79","Type":"ContainerStarted","Data":"c8ee6223f7790cd14c67b960c198d5635e386c79eef963927fe5cc401ab81af3"}
Sep 12 00:54:05.060915 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:05.060873    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd" podStartSLOduration=1.902078717 podStartE2EDuration="2.06085863s" totalImagesPullingTime="158.779913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:54:05.06044226 +0000 UTC m=+649.620893943" watchObservedRunningTime="2026-09-12 00:54:05.06085863 +0000 UTC m=+649.621310311"
Sep 12 00:54:12.944858 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:12.944816    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:54:13.761204 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:13.761170    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:13.761371 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:13.761230    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:13.773495 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:13.773474    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:14.091634 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:14.091557    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhpzd"
Sep 12 00:54:23.572182 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:23.572148    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:54:35.938151 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:35.938117    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"]
Sep 12 00:54:35.943859 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:35.943837    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:35.946443 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:35.946422    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 12 00:54:35.951560 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:35.951309    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"]
Sep 12 00:54:35.966941 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:35.966914    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:35.967090 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:35.966974    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:35.967090 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:35.966995    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:35.967090 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:35.967015    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dx6d\" (UniqueName: \"kubernetes.io/projected/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-kube-api-access-5dx6d\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:35.967274 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:35.967110    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:35.967274 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:35.967156    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:35.980742 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:35.980700    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:54:36.067608 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:36.067577    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:36.067794 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:36.067624    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:36.067794 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:36.067659    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:36.067794 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:36.067683    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dx6d\" (UniqueName: \"kubernetes.io/projected/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-kube-api-access-5dx6d\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:36.067794 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:36.067725    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:36.067794 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:36.067779    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:36.068096 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:36.068016    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:36.068148 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:36.068087    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:36.068250 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:36.068226    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:36.070364 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:36.070340    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:36.070669 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:36.070646    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:36.076198 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:36.076175    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dx6d\" (UniqueName: \"kubernetes.io/projected/70d9b75a-7b92-4dbe-afc1-aed86de0ec95-kube-api-access-5dx6d\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6f89h\" (UID: \"70d9b75a-7b92-4dbe-afc1-aed86de0ec95\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:36.254551 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:36.254482    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:36.528897 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:36.528873    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"]
Sep 12 00:54:36.531161 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:54:36.531136    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70d9b75a_7b92_4dbe_afc1_aed86de0ec95.slice/crio-f7eb393285699a44236eed0943384233cf2ceef0817b896398dd0fbd193be3aa WatchSource:0}: Error finding container f7eb393285699a44236eed0943384233cf2ceef0817b896398dd0fbd193be3aa: Status 404 returned error can't find the container with id f7eb393285699a44236eed0943384233cf2ceef0817b896398dd0fbd193be3aa
Sep 12 00:54:37.184020 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:37.183985    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h" event={"ID":"70d9b75a-7b92-4dbe-afc1-aed86de0ec95","Type":"ContainerStarted","Data":"dd3a978e8a9195b371eb8df42cbbecd8f6ee53ae3cf11c6f8f8458930780399d"}
Sep 12 00:54:37.184401 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:37.184034    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h" event={"ID":"70d9b75a-7b92-4dbe-afc1-aed86de0ec95","Type":"ContainerStarted","Data":"f7eb393285699a44236eed0943384233cf2ceef0817b896398dd0fbd193be3aa"}
Sep 12 00:54:37.201694 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:37.201645    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h" podStartSLOduration=1.9973971270000002 podStartE2EDuration="2.201630702s" totalImagesPullingTime="204.233575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:54:37.199908459 +0000 UTC m=+681.760360130" watchObservedRunningTime="2026-09-12 00:54:37.201630702 +0000 UTC m=+681.762082370"
Sep 12 00:54:46.255244 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:46.255207    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:46.255600 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:46.255471    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:46.267935 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:46.267910    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:54:47.241015 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:54:47.240986    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6f89h"
Sep 12 00:55:01.787328 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:55:01.787295    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:55:17.754343 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:55:17.754262    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:55:17.856007 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:55:17.855974    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:55:20.947121 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:55:20.947090    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:55:49.256327 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:55:49.256290    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:55:49.351654 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:55:49.351616    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:55:49.713578 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:55:49.713533    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:55:50.445193 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:55:50.445162    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:55:50.615009 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:55:50.614977    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:55:52.887203 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:55:52.887167    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:55:52.988264 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:55:52.988236    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:06.046549 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:06.046517    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:06.149220 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:06.149182    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:06.248073 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:06.248043    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:06.411712 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:06.411667    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:06.606604 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:06.606528    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:06.723239 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:06.723121    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:06.822941 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:06.822898    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:07.112225 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:07.112112    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:07.561869 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:07.561836    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:07.650890 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:07.650859    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:08.045517 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:08.045488    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:08.158174 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:08.158143    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8032-2222ee"]
Sep 12 00:56:08.161370 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:08.161354    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8032-2222ee"
Sep 12 00:56:08.163747 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:08.163731    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n87bf\""
Sep 12 00:56:08.169169 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:08.169145    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8032-2222ee"]
Sep 12 00:56:08.306081 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:08.306001    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhnss\" (UniqueName: \"kubernetes.io/projected/7b94018a-c6eb-45ce-99ed-e09c01a56abc-kube-api-access-vhnss\") pod \"test-curl-8032-2222ee\" (UID: \"7b94018a-c6eb-45ce-99ed-e09c01a56abc\") " pod="opendatahub/test-curl-8032-2222ee"
Sep 12 00:56:08.407066 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:08.407031    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhnss\" (UniqueName: \"kubernetes.io/projected/7b94018a-c6eb-45ce-99ed-e09c01a56abc-kube-api-access-vhnss\") pod \"test-curl-8032-2222ee\" (UID: \"7b94018a-c6eb-45ce-99ed-e09c01a56abc\") " pod="opendatahub/test-curl-8032-2222ee"
Sep 12 00:56:08.416749 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:08.416719    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhnss\" (UniqueName: \"kubernetes.io/projected/7b94018a-c6eb-45ce-99ed-e09c01a56abc-kube-api-access-vhnss\") pod \"test-curl-8032-2222ee\" (UID: \"7b94018a-c6eb-45ce-99ed-e09c01a56abc\") " pod="opendatahub/test-curl-8032-2222ee"
Sep 12 00:56:08.471198 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:08.471164    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8032-2222ee"
Sep 12 00:56:08.742183 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:08.742147    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8032-2222ee"]
Sep 12 00:56:08.750359 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:56:08.750333    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b94018a_c6eb_45ce_99ed_e09c01a56abc.slice/crio-a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7 WatchSource:0}: Error finding container a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7: Status 404 returned error can't find the container with id a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7
Sep 12 00:56:08.826867 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:08.826836    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:09.605060 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:09.605009    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8032-2222ee" event={"ID":"7b94018a-c6eb-45ce-99ed-e09c01a56abc","Type":"ContainerStarted","Data":"a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7"}
Sep 12 00:56:09.607505 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:09.607474    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:09.699883 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:09.699854    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:09.824238 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:09.824193    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:10.048627 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:10.048572    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:11.278964 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.278912    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"]
Sep 12 00:56:11.282808 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.282786    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:11.285206 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.285185    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-f19fd1-serving-cert\""
Sep 12 00:56:11.294659 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.294636    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"]
Sep 12 00:56:11.314493 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.314471    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:11.329494 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.329473    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-metrics-certs\") pod \"maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:11.329600 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.329500    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/778eaff8-6d09-41d7-812f-7ab5a2e4636b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:11.329600 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.329572    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnkn6\" (UniqueName: \"kubernetes.io/projected/778eaff8-6d09-41d7-812f-7ab5a2e4636b-kube-api-access-fnkn6\") pod \"maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:11.329719 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.329642    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:11.390503 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.390473    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:11.430660 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.430631    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-metrics-certs\") pod \"maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:11.430784 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.430670    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/778eaff8-6d09-41d7-812f-7ab5a2e4636b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:11.430784 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.430732    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnkn6\" (UniqueName: \"kubernetes.io/projected/778eaff8-6d09-41d7-812f-7ab5a2e4636b-kube-api-access-fnkn6\") pod \"maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:11.430890 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.430804    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:11.430976 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:11.430926    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-serving-cert: secret "maas-api-e2e-worker-w4-f19fd1-serving-cert" not found
Sep 12 00:56:11.431038 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:11.431021    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-maas-api-tls podName:778eaff8-6d09-41d7-812f-7ab5a2e4636b nodeName:}" failed. No retries permitted until 2026-09-12 00:56:11.930999783 +0000 UTC m=+776.491451449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-maas-api-tls") pod "maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh" (UID: "778eaff8-6d09-41d7-812f-7ab5a2e4636b") : secret "maas-api-e2e-worker-w4-f19fd1-serving-cert" not found
Sep 12 00:56:11.431419 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.431402    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/778eaff8-6d09-41d7-812f-7ab5a2e4636b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:11.433254 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.433233    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-metrics-certs\") pod \"maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:11.442802 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.442783    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnkn6\" (UniqueName: \"kubernetes.io/projected/778eaff8-6d09-41d7-812f-7ab5a2e4636b-kube-api-access-fnkn6\") pod \"maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:11.610823 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.610738    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:11.615741 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.615718    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8032-2222ee" event={"ID":"7b94018a-c6eb-45ce-99ed-e09c01a56abc","Type":"ContainerStarted","Data":"675af6f41892601013b704fbaf465df06e0cebffd9a74e0e722882ceb294979d"}
Sep 12 00:56:11.638226 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.638187    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8032-2222ee" podStartSLOduration=1.621997234 podStartE2EDuration="3.638173533s" totalImagesPullingTime="2.016176299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:11.63725373 +0000 UTC m=+776.197705403" watchObservedRunningTime="2026-09-12 00:56:11.638173533 +0000 UTC m=+776.198625201"
Sep 12 00:56:11.847380 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.847356    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:11.933838 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.933804    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:11.936267 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.936247    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:11.943726 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:11.943700    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:12.194516 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.194449    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:12.325804 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.325771    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8032-2222ee"]
Sep 12 00:56:12.337476 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.337448    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8032-2222ee"]
Sep 12 00:56:12.449800 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.449731    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8032-c5d604"]
Sep 12 00:56:12.450130 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.450116    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b94018a-c6eb-45ce-99ed-e09c01a56abc" containerName="test-curl-8032-2222ee"
Sep 12 00:56:12.450181 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.450132    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b94018a-c6eb-45ce-99ed-e09c01a56abc" containerName="test-curl-8032-2222ee"
Sep 12 00:56:12.450214 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.450198    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b94018a-c6eb-45ce-99ed-e09c01a56abc" containerName="test-curl-8032-2222ee"
Sep 12 00:56:12.454531 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.454506    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8032-c5d604"
Sep 12 00:56:12.463426 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.463403    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8032-c5d604"]
Sep 12 00:56:12.467992 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.467969    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"]
Sep 12 00:56:12.471305 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.471287    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:12.474404 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.474388    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5cee89-serving-cert\""
Sep 12 00:56:12.483253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.482281    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"]
Sep 12 00:56:12.484428 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:56:12.484108    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778eaff8_6d09_41d7_812f_7ab5a2e4636b.slice/crio-dec5067e7be3fe13aefd573032b1451ba15b54ea62f2d81bfb22755d7e51be0e WatchSource:0}: Error finding container dec5067e7be3fe13aefd573032b1451ba15b54ea62f2d81bfb22755d7e51be0e: Status 404 returned error can't find the container with id dec5067e7be3fe13aefd573032b1451ba15b54ea62f2d81bfb22755d7e51be0e
Sep 12 00:56:12.489356 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.489229    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"]
Sep 12 00:56:12.512571 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.512534    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:12.539915 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.539887    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-metrics-certs\") pod \"maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:12.540071 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.539976    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-maas-api-tls\") pod \"maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:12.540071 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.540018    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bbe704-fc50-475e-ada4-5bf62d5bb62d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:12.540071 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.540046    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24xht\" (UniqueName: \"kubernetes.io/projected/21bbe704-fc50-475e-ada4-5bf62d5bb62d-kube-api-access-24xht\") pod \"maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:12.540241 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.540073    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvgwn\" (UniqueName: \"kubernetes.io/projected/e2e12285-aa70-4530-91d8-682fc3aebcf0-kube-api-access-mvgwn\") pod \"test-curl-8032-c5d604\" (UID: \"e2e12285-aa70-4530-91d8-682fc3aebcf0\") " pod="opendatahub/test-curl-8032-c5d604"
Sep 12 00:56:12.586723 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.586666    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:12.628564 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.628530    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh" event={"ID":"778eaff8-6d09-41d7-812f-7ab5a2e4636b","Type":"ContainerStarted","Data":"dec5067e7be3fe13aefd573032b1451ba15b54ea62f2d81bfb22755d7e51be0e"}
Sep 12 00:56:12.628830 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:12.628809    2567 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8032-2222ee" container="test-curl-8032-2222ee"
Sep 12 00:56:12.640941 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.640915    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-maas-api-tls\") pod \"maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:12.641065 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.640964    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bbe704-fc50-475e-ada4-5bf62d5bb62d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:12.641065 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.640983    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24xht\" (UniqueName: \"kubernetes.io/projected/21bbe704-fc50-475e-ada4-5bf62d5bb62d-kube-api-access-24xht\") pod \"maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:12.641065 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.641015    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvgwn\" (UniqueName: \"kubernetes.io/projected/e2e12285-aa70-4530-91d8-682fc3aebcf0-kube-api-access-mvgwn\") pod \"test-curl-8032-c5d604\" (UID: \"e2e12285-aa70-4530-91d8-682fc3aebcf0\") " pod="opendatahub/test-curl-8032-c5d604"
Sep 12 00:56:12.641230 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:12.641066    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-serving-cert: secret "maas-api-e2e-inf-a-5cee89-serving-cert" not found
Sep 12 00:56:12.641230 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:12.641134    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-maas-api-tls podName:21bbe704-fc50-475e-ada4-5bf62d5bb62d nodeName:}" failed. No retries permitted until 2026-09-12 00:56:13.14111183 +0000 UTC m=+777.701563490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-maas-api-tls") pod "maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt" (UID: "21bbe704-fc50-475e-ada4-5bf62d5bb62d") : secret "maas-api-e2e-inf-a-5cee89-serving-cert" not found
Sep 12 00:56:12.641230 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.641066    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-metrics-certs\") pod \"maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:12.641698 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.641679    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bbe704-fc50-475e-ada4-5bf62d5bb62d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:12.643662 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.643647    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-metrics-certs\") pod \"maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:12.650721 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.650679    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24xht\" (UniqueName: \"kubernetes.io/projected/21bbe704-fc50-475e-ada4-5bf62d5bb62d-kube-api-access-24xht\") pod \"maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:12.650917 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.650899    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvgwn\" (UniqueName: \"kubernetes.io/projected/e2e12285-aa70-4530-91d8-682fc3aebcf0-kube-api-access-mvgwn\") pod \"test-curl-8032-c5d604\" (UID: \"e2e12285-aa70-4530-91d8-682fc3aebcf0\") " pod="opendatahub/test-curl-8032-c5d604"
Sep 12 00:56:12.768604 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.768542    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8032-c5d604"
Sep 12 00:56:12.968610 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:12.968577    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:13.044735 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.044717    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8032-c5d604"]
Sep 12 00:56:13.046936 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:56:13.046905    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2e12285_aa70_4530_91d8_682fc3aebcf0.slice/crio-6cae177d78c2a53e850f1dc03fa5cca467d3ac1fa9c719e94bd7d2f79e0398b2 WatchSource:0}: Error finding container 6cae177d78c2a53e850f1dc03fa5cca467d3ac1fa9c719e94bd7d2f79e0398b2: Status 404 returned error can't find the container with id 6cae177d78c2a53e850f1dc03fa5cca467d3ac1fa9c719e94bd7d2f79e0398b2
Sep 12 00:56:13.144101 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.144073    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-maas-api-tls\") pod \"maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:13.146866 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.146834    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-maas-api-tls\") pod \"maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:13.166054 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.166027    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:13.251189 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.251148    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:13.385467 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.385398    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:13.635539 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.635437    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh" event={"ID":"778eaff8-6d09-41d7-812f-7ab5a2e4636b","Type":"ContainerStarted","Data":"3a71b6436514bb569fdab24afc141165b459b69936ae9d84f34af57b8b70dbba"}
Sep 12 00:56:13.635704 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.635594    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:13.637321 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.637291    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8032-c5d604" event={"ID":"e2e12285-aa70-4530-91d8-682fc3aebcf0","Type":"ContainerStarted","Data":"aa72e12f5425d444a77abb02c0fa4b4bbedf6ca4a8397f0f63bd456797ad9bb6"}
Sep 12 00:56:13.637433 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.637331    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8032-c5d604" event={"ID":"e2e12285-aa70-4530-91d8-682fc3aebcf0","Type":"ContainerStarted","Data":"6cae177d78c2a53e850f1dc03fa5cca467d3ac1fa9c719e94bd7d2f79e0398b2"}
Sep 12 00:56:13.637433 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.637357    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8032-2222ee" podUID="7b94018a-c6eb-45ce-99ed-e09c01a56abc" containerName="test-curl-8032-2222ee" containerID="cri-o://675af6f41892601013b704fbaf465df06e0cebffd9a74e0e722882ceb294979d" gracePeriod=2
Sep 12 00:56:13.655026 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.654976    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh" podStartSLOduration=2.198398438 podStartE2EDuration="2.65492844s" totalImagesPullingTime="456.530002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:13.654328733 +0000 UTC m=+778.214780423" watchObservedRunningTime="2026-09-12 00:56:13.65492844 +0000 UTC m=+778.215380110"
Sep 12 00:56:13.656343 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:13.656312    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8032-2222ee\" is forbidden: User \"system:node:ip-10-0-139-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-121.ec2.internal' and this object" podUID="7b94018a-c6eb-45ce-99ed-e09c01a56abc" pod="opendatahub/test-curl-8032-2222ee"
Sep 12 00:56:13.660964 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.660929    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"]
Sep 12 00:56:13.666481 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.666436    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"]
Sep 12 00:56:13.666600 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.666536    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:13.667180 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:56:13.667156    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bbe704_fc50_475e_ada4_5bf62d5bb62d.slice/crio-7f777a49a6164291ad212df5d3f4e288dacdc21062557c3b1193f09e88a98570 WatchSource:0}: Error finding container 7f777a49a6164291ad212df5d3f4e288dacdc21062557c3b1193f09e88a98570: Status 404 returned error can't find the container with id 7f777a49a6164291ad212df5d3f4e288dacdc21062557c3b1193f09e88a98570
Sep 12 00:56:13.668917 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.668897    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-10ba6f-serving-cert\""
Sep 12 00:56:13.677067 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.676625    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"]
Sep 12 00:56:13.683748 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.683710    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8032-c5d604" podStartSLOduration=1.683696432 podStartE2EDuration="1.683696432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:13.682586954 +0000 UTC m=+778.243038623" watchObservedRunningTime="2026-09-12 00:56:13.683696432 +0000 UTC m=+778.244148100"
Sep 12 00:56:13.701872 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:13.701840    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8032-2222ee\" is forbidden: User \"system:node:ip-10-0-139-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-121.ec2.internal' and this object" podUID="7b94018a-c6eb-45ce-99ed-e09c01a56abc" pod="opendatahub/test-curl-8032-2222ee"
Sep 12 00:56:13.711199 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.711173    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:13.748484 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.748457    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a35c6d-185e-485a-bece-6da769a0f541-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:13.748579 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.748489    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49dnl\" (UniqueName: \"kubernetes.io/projected/e8a35c6d-185e-485a-bece-6da769a0f541-kube-api-access-49dnl\") pod \"maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:13.748579 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.748565    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-maas-api-tls\") pod \"maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:13.748651 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.748625    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-metrics-certs\") pod \"maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:13.775474 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.775449    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:13.849253 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.849225    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-maas-api-tls\") pod \"maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:13.849364 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.849289    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-metrics-certs\") pod \"maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:13.849364 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.849341    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a35c6d-185e-485a-bece-6da769a0f541-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:13.849476 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.849371    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49dnl\" (UniqueName: \"kubernetes.io/projected/e8a35c6d-185e-485a-bece-6da769a0f541-kube-api-access-49dnl\") pod \"maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:13.849476 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:13.849386    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-serving-cert: secret "maas-api-e2e-worker-w0-10ba6f-serving-cert" not found
Sep 12 00:56:13.849476 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:13.849451    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-maas-api-tls podName:e8a35c6d-185e-485a-bece-6da769a0f541 nodeName:}" failed. No retries permitted until 2026-09-12 00:56:14.34943115 +0000 UTC m=+778.909882815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-maas-api-tls") pod "maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n" (UID: "e8a35c6d-185e-485a-bece-6da769a0f541") : secret "maas-api-e2e-worker-w0-10ba6f-serving-cert" not found
Sep 12 00:56:13.850034 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.850016    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a35c6d-185e-485a-bece-6da769a0f541-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:13.851896 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.851879    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-metrics-certs\") pod \"maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:13.861612 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.861589    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49dnl\" (UniqueName: \"kubernetes.io/projected/e8a35c6d-185e-485a-bece-6da769a0f541-kube-api-access-49dnl\") pod \"maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:13.990085 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:13.990059    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:14.201162 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:14.201133    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:14.281447 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:14.281365    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8032-c5d604"]
Sep 12 00:56:14.285821 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:14.285785    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:14.287715 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:14.287696    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8032-c5d604"]
Sep 12 00:56:14.353915 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:14.353865    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-maas-api-tls\") pod \"maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:14.356505 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:14.356479    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-maas-api-tls\") pod \"maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:14.580720 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:14.580642    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:14.612782 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:14.612752    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:14.645508 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:14.645479    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt" event={"ID":"21bbe704-fc50-475e-ada4-5bf62d5bb62d","Type":"ContainerStarted","Data":"4eb0f4d4e65f1c1754ad25b37c3329c6e3d9a2e2e9c87f94cda5d3522f31117f"}
Sep 12 00:56:14.645633 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:14.645534    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt" event={"ID":"21bbe704-fc50-475e-ada4-5bf62d5bb62d","Type":"ContainerStarted","Data":"7f777a49a6164291ad212df5d3f4e288dacdc21062557c3b1193f09e88a98570"}
Sep 12 00:56:14.645734 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:14.645719    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:14.645961 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:14.645934    2567 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8032-c5d604" container="test-curl-8032-c5d604"
Sep 12 00:56:14.664049 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:14.663990    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt" podStartSLOduration=2.311478965 podStartE2EDuration="2.663974452s" totalImagesPullingTime="352.495487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:14.663302368 +0000 UTC m=+779.223754062" watchObservedRunningTime="2026-09-12 00:56:14.663974452 +0000 UTC m=+779.224426118"
Sep 12 00:56:14.800360 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:14.800319    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:14.854861 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:14.854840    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"]
Sep 12 00:56:14.856433 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:56:14.856412    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a35c6d_185e_485a_bece_6da769a0f541.slice/crio-791d8cc176280568fcbd715eb7d11f14a52147d8bc9122d035ce601d8d57f51d WatchSource:0}: Error finding container 791d8cc176280568fcbd715eb7d11f14a52147d8bc9122d035ce601d8d57f51d: Status 404 returned error can't find the container with id 791d8cc176280568fcbd715eb7d11f14a52147d8bc9122d035ce601d8d57f51d
Sep 12 00:56:15.092581 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:15.092547    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:15.479119 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:15.479089    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:15.600479 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:15.600442    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:15.658834 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:15.658795    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n" event={"ID":"e8a35c6d-185e-485a-bece-6da769a0f541","Type":"ContainerStarted","Data":"fb4c9c01f7666c5cfe76ea6441792a091554e3271e868504d534ffeeb6ab3267"}
Sep 12 00:56:15.659023 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:15.658898    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n" event={"ID":"e8a35c6d-185e-485a-bece-6da769a0f541","Type":"ContainerStarted","Data":"791d8cc176280568fcbd715eb7d11f14a52147d8bc9122d035ce601d8d57f51d"}
Sep 12 00:56:15.659149 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:15.659120    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8032-c5d604" podUID="e2e12285-aa70-4530-91d8-682fc3aebcf0" containerName="test-curl-8032-c5d604" containerID="cri-o://aa72e12f5425d444a77abb02c0fa4b4bbedf6ca4a8397f0f63bd456797ad9bb6" gracePeriod=2
Sep 12 00:56:15.659539 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:15.659127    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:15.677196 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:15.677150    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n" podStartSLOduration=2.085903733 podStartE2EDuration="2.677131018s" totalImagesPullingTime="591.227285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:15.676605217 +0000 UTC m=+780.237056925" watchObservedRunningTime="2026-09-12 00:56:15.677131018 +0000 UTC m=+780.237582687"
Sep 12 00:56:15.678777 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:15.678754    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8032-c5d604\" is forbidden: User \"system:node:ip-10-0-139-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-121.ec2.internal' and this object" podUID="e2e12285-aa70-4530-91d8-682fc3aebcf0" pod="opendatahub/test-curl-8032-c5d604"
Sep 12 00:56:15.823172 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:15.823145    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8032-2222ee"
Sep 12 00:56:15.825208 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:15.825186    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8032-c5d604\" is forbidden: User \"system:node:ip-10-0-139-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-121.ec2.internal' and this object" podUID="e2e12285-aa70-4530-91d8-682fc3aebcf0" pod="opendatahub/test-curl-8032-c5d604"
Sep 12 00:56:15.868460 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:15.868436    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhnss\" (UniqueName: \"kubernetes.io/projected/7b94018a-c6eb-45ce-99ed-e09c01a56abc-kube-api-access-vhnss\") pod \"7b94018a-c6eb-45ce-99ed-e09c01a56abc\" (UID: \"7b94018a-c6eb-45ce-99ed-e09c01a56abc\") "
Sep 12 00:56:15.870567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:15.870541    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b94018a-c6eb-45ce-99ed-e09c01a56abc-kube-api-access-vhnss" (OuterVolumeSpecName: "kube-api-access-vhnss") pod "7b94018a-c6eb-45ce-99ed-e09c01a56abc" (UID: "7b94018a-c6eb-45ce-99ed-e09c01a56abc"). InnerVolumeSpecName "kube-api-access-vhnss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:56:15.897309 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:15.897275    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:15.969826 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:15.969795    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhnss\" (UniqueName: \"kubernetes.io/projected/7b94018a-c6eb-45ce-99ed-e09c01a56abc-kube-api-access-vhnss\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:56:15.973573 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:15.973541    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8032-c5d604\" is forbidden: User \"system:node:ip-10-0-139-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-121.ec2.internal' and this object" podUID="e2e12285-aa70-4530-91d8-682fc3aebcf0" pod="opendatahub/test-curl-8032-c5d604"
Sep 12 00:56:15.974493 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:15.974474    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b94018a-c6eb-45ce-99ed-e09c01a56abc" path="/var/lib/kubelet/pods/7b94018a-c6eb-45ce-99ed-e09c01a56abc/volumes"
Sep 12 00:56:16.004498 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:16.004419    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:16.203124 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:16.203076    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:16.537206 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:16.537167    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:16.644578 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:16.644550    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:16.665361 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:16.665333    2567 generic.go:356] "Generic (PLEG): container finished" podID="7b94018a-c6eb-45ce-99ed-e09c01a56abc" containerID="675af6f41892601013b704fbaf465df06e0cebffd9a74e0e722882ceb294979d" exitCode=137
Sep 12 00:56:16.665494 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:16.665384    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8032-2222ee"
Sep 12 00:56:16.665494 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:16.665424    2567 scope.go:117] "RemoveContainer" containerID="675af6f41892601013b704fbaf465df06e0cebffd9a74e0e722882ceb294979d"
Sep 12 00:56:16.673994 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:16.673978    2567 scope.go:117] "RemoveContainer" containerID="675af6f41892601013b704fbaf465df06e0cebffd9a74e0e722882ceb294979d"
Sep 12 00:56:16.674270 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:16.674253    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"675af6f41892601013b704fbaf465df06e0cebffd9a74e0e722882ceb294979d\": container with ID starting with 675af6f41892601013b704fbaf465df06e0cebffd9a74e0e722882ceb294979d not found: ID does not exist" containerID="675af6f41892601013b704fbaf465df06e0cebffd9a74e0e722882ceb294979d"
Sep 12 00:56:16.674327 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:16.674283    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"675af6f41892601013b704fbaf465df06e0cebffd9a74e0e722882ceb294979d"} err="failed to get container status \"675af6f41892601013b704fbaf465df06e0cebffd9a74e0e722882ceb294979d\": rpc error: code = NotFound desc = could not find container \"675af6f41892601013b704fbaf465df06e0cebffd9a74e0e722882ceb294979d\": container with ID starting with 675af6f41892601013b704fbaf465df06e0cebffd9a74e0e722882ceb294979d not found: ID does not exist"
Sep 12 00:56:16.857341 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:16.857256    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:17.587093 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:17.587062    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:17.854110 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:17.854086    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8032-c5d604"
Sep 12 00:56:17.874051 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:17.874020    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:17.886468 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:17.886434    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvgwn\" (UniqueName: \"kubernetes.io/projected/e2e12285-aa70-4530-91d8-682fc3aebcf0-kube-api-access-mvgwn\") pod \"e2e12285-aa70-4530-91d8-682fc3aebcf0\" (UID: \"e2e12285-aa70-4530-91d8-682fc3aebcf0\") "
Sep 12 00:56:17.888631 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:17.888602    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e12285-aa70-4530-91d8-682fc3aebcf0-kube-api-access-mvgwn" (OuterVolumeSpecName: "kube-api-access-mvgwn") pod "e2e12285-aa70-4530-91d8-682fc3aebcf0" (UID: "e2e12285-aa70-4530-91d8-682fc3aebcf0"). InnerVolumeSpecName "kube-api-access-mvgwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:56:17.946396 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:17.946365    2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b94018a_c6eb_45ce_99ed_e09c01a56abc.slice/crio-a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7\": RecentStats: unable to find data in memory cache]"
Sep 12 00:56:17.973291 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:17.973269    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2e12285-aa70-4530-91d8-682fc3aebcf0" path="/var/lib/kubelet/pods/e2e12285-aa70-4530-91d8-682fc3aebcf0/volumes"
Sep 12 00:56:17.982235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:17.982218    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:17.988011 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:17.987996    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvgwn\" (UniqueName: \"kubernetes.io/projected/e2e12285-aa70-4530-91d8-682fc3aebcf0-kube-api-access-mvgwn\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:56:18.677227 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:18.677195    2567 generic.go:356] "Generic (PLEG): container finished" podID="e2e12285-aa70-4530-91d8-682fc3aebcf0" containerID="aa72e12f5425d444a77abb02c0fa4b4bbedf6ca4a8397f0f63bd456797ad9bb6" exitCode=137
Sep 12 00:56:18.677377 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:18.677233    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8032-c5d604"
Sep 12 00:56:18.677377 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:18.677298    2567 scope.go:117] "RemoveContainer" containerID="aa72e12f5425d444a77abb02c0fa4b4bbedf6ca4a8397f0f63bd456797ad9bb6"
Sep 12 00:56:18.685590 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:18.685571    2567 scope.go:117] "RemoveContainer" containerID="aa72e12f5425d444a77abb02c0fa4b4bbedf6ca4a8397f0f63bd456797ad9bb6"
Sep 12 00:56:18.685876 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:18.685859    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa72e12f5425d444a77abb02c0fa4b4bbedf6ca4a8397f0f63bd456797ad9bb6\": container with ID starting with aa72e12f5425d444a77abb02c0fa4b4bbedf6ca4a8397f0f63bd456797ad9bb6 not found: ID does not exist" containerID="aa72e12f5425d444a77abb02c0fa4b4bbedf6ca4a8397f0f63bd456797ad9bb6"
Sep 12 00:56:18.685925 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:18.685883    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa72e12f5425d444a77abb02c0fa4b4bbedf6ca4a8397f0f63bd456797ad9bb6"} err="failed to get container status \"aa72e12f5425d444a77abb02c0fa4b4bbedf6ca4a8397f0f63bd456797ad9bb6\": rpc error: code = NotFound desc = could not find container \"aa72e12f5425d444a77abb02c0fa4b4bbedf6ca4a8397f0f63bd456797ad9bb6\": container with ID starting with aa72e12f5425d444a77abb02c0fa4b4bbedf6ca4a8397f0f63bd456797ad9bb6 not found: ID does not exist"
Sep 12 00:56:18.922729 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:18.922694    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:19.349967 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:19.349919    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:19.458699 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:19.458665    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:19.650984 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:19.650889    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 00:56:19.987552 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:19.987511    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:20.664486 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:20.664456    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 00:56:21.204992 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:21.204940    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:21.670943 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:21.670916    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 00:56:22.298964 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.298923    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"]
Sep 12 00:56:22.299525 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.299506    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2e12285-aa70-4530-91d8-682fc3aebcf0" containerName="test-curl-8032-c5d604"
Sep 12 00:56:22.299617 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.299528    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2e12285-aa70-4530-91d8-682fc3aebcf0" containerName="test-curl-8032-c5d604"
Sep 12 00:56:22.299680 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.299641    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2e12285-aa70-4530-91d8-682fc3aebcf0" containerName="test-curl-8032-c5d604"
Sep 12 00:56:22.304607 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.304582    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:22.307402 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.307377    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-64f698-serving-cert\""
Sep 12 00:56:22.311727 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.311702    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"]
Sep 12 00:56:22.329641 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.329620    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72b1b072-5cd0-4677-b516-8cd0b234ad2e-maas-api-tls\") pod \"maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:22.329787 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.329673    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72b1b072-5cd0-4677-b516-8cd0b234ad2e-openshift-service-ca\") pod \"maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:22.329787 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.329693    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zq8j\" (UniqueName: \"kubernetes.io/projected/72b1b072-5cd0-4677-b516-8cd0b234ad2e-kube-api-access-5zq8j\") pod \"maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:22.329892 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.329779    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72b1b072-5cd0-4677-b516-8cd0b234ad2e-metrics-certs\") pod \"maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:22.330728 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.330702    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:22.430311 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.430282    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72b1b072-5cd0-4677-b516-8cd0b234ad2e-maas-api-tls\") pod \"maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:22.430536 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.430330    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72b1b072-5cd0-4677-b516-8cd0b234ad2e-openshift-service-ca\") pod \"maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:22.430536 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.430349    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zq8j\" (UniqueName: \"kubernetes.io/projected/72b1b072-5cd0-4677-b516-8cd0b234ad2e-kube-api-access-5zq8j\") pod \"maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:22.430536 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.430366    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72b1b072-5cd0-4677-b516-8cd0b234ad2e-metrics-certs\") pod \"maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:22.430536 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.430517    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:22.431016 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.430998    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72b1b072-5cd0-4677-b516-8cd0b234ad2e-openshift-service-ca\") pod \"maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:22.433032 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.433014    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72b1b072-5cd0-4677-b516-8cd0b234ad2e-metrics-certs\") pod \"maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:22.433175 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.433157    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72b1b072-5cd0-4677-b516-8cd0b234ad2e-maas-api-tls\") pod \"maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:22.438437 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.438410    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zq8j\" (UniqueName: \"kubernetes.io/projected/72b1b072-5cd0-4677-b516-8cd0b234ad2e-kube-api-access-5zq8j\") pod \"maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:22.619650 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.619580    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:22.888309 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:22.888288    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"]
Sep 12 00:56:22.890524 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:56:22.890499    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b1b072_5cd0_4677_b516_8cd0b234ad2e.slice/crio-bea8597d08adcb83539e662e05e5b064870d5ff86e25ad8f403e3cadb80291af WatchSource:0}: Error finding container bea8597d08adcb83539e662e05e5b064870d5ff86e25ad8f403e3cadb80291af: Status 404 returned error can't find the container with id bea8597d08adcb83539e662e05e5b064870d5ff86e25ad8f403e3cadb80291af
Sep 12 00:56:23.028830 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.028796    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:23.129495 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.129437    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:23.704482 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.704442    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9" event={"ID":"72b1b072-5cd0-4677-b516-8cd0b234ad2e","Type":"ContainerStarted","Data":"bc7946b662bf8982a0df0f2a85e3318aa1a14dca6e60830a7ab2ed79fe78529a"}
Sep 12 00:56:23.704972 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.704498    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9" event={"ID":"72b1b072-5cd0-4677-b516-8cd0b234ad2e","Type":"ContainerStarted","Data":"bea8597d08adcb83539e662e05e5b064870d5ff86e25ad8f403e3cadb80291af"}
Sep 12 00:56:23.704972 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.704572    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:23.718508 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.718480    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"]
Sep 12 00:56:23.721971 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.721930    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:23.724452 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.724427    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b2dc0e67-serving-cert\""
Sep 12 00:56:23.724754 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.724716    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9" podStartSLOduration=1.334049097 podStartE2EDuration="1.724702993s" totalImagesPullingTime="390.653896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:23.723321643 +0000 UTC m=+788.283773322" watchObservedRunningTime="2026-09-12 00:56:23.724702993 +0000 UTC m=+788.285154661"
Sep 12 00:56:23.732974 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.732888    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"]
Sep 12 00:56:23.758238 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.758201    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:23.822942 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.822914    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:23.843871 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.843843    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352b38fc-c8de-451c-a8e2-23b20c33315d-maas-api-tls\") pod \"maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:23.844016 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.843922    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352b38fc-c8de-451c-a8e2-23b20c33315d-openshift-service-ca\") pod \"maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:23.844016 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.843977    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjzhf\" (UniqueName: \"kubernetes.io/projected/352b38fc-c8de-451c-a8e2-23b20c33315d-kube-api-access-vjzhf\") pod \"maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:23.844100 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.844058    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352b38fc-c8de-451c-a8e2-23b20c33315d-metrics-certs\") pod \"maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:23.887489 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.887462    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:23.944460 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.944431    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352b38fc-c8de-451c-a8e2-23b20c33315d-openshift-service-ca\") pod \"maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:23.944591 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.944472    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjzhf\" (UniqueName: \"kubernetes.io/projected/352b38fc-c8de-451c-a8e2-23b20c33315d-kube-api-access-vjzhf\") pod \"maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:23.944660 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.944588    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352b38fc-c8de-451c-a8e2-23b20c33315d-metrics-certs\") pod \"maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:23.944721 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.944664    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352b38fc-c8de-451c-a8e2-23b20c33315d-maas-api-tls\") pod \"maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:23.945042 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.945022    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352b38fc-c8de-451c-a8e2-23b20c33315d-openshift-service-ca\") pod \"maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:23.947188 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.947167    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352b38fc-c8de-451c-a8e2-23b20c33315d-metrics-certs\") pod \"maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:23.947269 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.947170    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352b38fc-c8de-451c-a8e2-23b20c33315d-maas-api-tls\") pod \"maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:23.952525 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:23.952504    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjzhf\" (UniqueName: \"kubernetes.io/projected/352b38fc-c8de-451c-a8e2-23b20c33315d-kube-api-access-vjzhf\") pod \"maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:24.037047 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:24.036941    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:24.309725 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:24.309704    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"]
Sep 12 00:56:24.311580 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:56:24.311551    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352b38fc_c8de_451c_a8e2_23b20c33315d.slice/crio-3dd876ea1655da75d7bf2a211b4400e68e5ac66c94b7006f4fb165978a93ec6d WatchSource:0}: Error finding container 3dd876ea1655da75d7bf2a211b4400e68e5ac66c94b7006f4fb165978a93ec6d: Status 404 returned error can't find the container with id 3dd876ea1655da75d7bf2a211b4400e68e5ac66c94b7006f4fb165978a93ec6d
Sep 12 00:56:24.341970 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:24.341926    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:24.504589 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:24.504555    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:24.604961 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:24.604921    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:24.693588 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:24.693551    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:24.712268 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:24.712235    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq" event={"ID":"352b38fc-c8de-451c-a8e2-23b20c33315d","Type":"ContainerStarted","Data":"4ad185596154f1722b9b482254834d8456cb8d07df10e3aa51ef97ecd7a9725e"}
Sep 12 00:56:24.712664 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:24.712322    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq" event={"ID":"352b38fc-c8de-451c-a8e2-23b20c33315d","Type":"ContainerStarted","Data":"3dd876ea1655da75d7bf2a211b4400e68e5ac66c94b7006f4fb165978a93ec6d"}
Sep 12 00:56:24.712664 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:24.712396    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:24.737800 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:24.737751    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq" podStartSLOduration=1.4491301349999999 podStartE2EDuration="1.73773582s" totalImagesPullingTime="288.605685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:24.735998376 +0000 UTC m=+789.296450064" watchObservedRunningTime="2026-09-12 00:56:24.73773582 +0000 UTC m=+789.298187491"
Sep 12 00:56:24.783974 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:24.783924    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:24.878650 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:24.878574    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:24.986641 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:24.986595    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:25.099152 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:25.099102    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:25.312231 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:25.312195    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:25.485288 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:25.485262    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:26.402045 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:26.402020    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:27.479170 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:27.479138    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:27.966807 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:27.966764    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:27.994329 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:27.994303    2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b94018a_c6eb_45ce_99ed_e09c01a56abc.slice/crio-a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7\": RecentStats: unable to find data in memory cache]"
Sep 12 00:56:28.055357 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:28.055332    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:28.553209 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:28.553171    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:29.428201 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:29.428168    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:29.527922 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:29.527892    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:29.629837 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:29.629801    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:29.718632 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:29.718559    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:56:29.872569 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:29.872529    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:29.976505 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:29.976377    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:30.073491 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:30.073461    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:30.722921 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:30.722894    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:30.753291 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:30.753263    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:31.761618 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.761587    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"]
Sep 12 00:56:31.765348 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.765331    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:31.768066 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.768046    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-bf547a-serving-cert\""
Sep 12 00:56:31.777883 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.777861    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"]
Sep 12 00:56:31.823910 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.823880    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf87a86-35e2-449e-8e51-f93ca5c133d3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:31.824103 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.823921    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qcqz\" (UniqueName: \"kubernetes.io/projected/bbf87a86-35e2-449e-8e51-f93ca5c133d3-kube-api-access-6qcqz\") pod \"maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:31.824103 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.824003    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf87a86-35e2-449e-8e51-f93ca5c133d3-maas-api-tls\") pod \"maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:31.824103 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.824084    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf87a86-35e2-449e-8e51-f93ca5c133d3-metrics-certs\") pod \"maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:31.855611 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.855576    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:31.925451 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.925416    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf87a86-35e2-449e-8e51-f93ca5c133d3-maas-api-tls\") pod \"maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:31.925619 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.925514    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf87a86-35e2-449e-8e51-f93ca5c133d3-metrics-certs\") pod \"maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:31.925619 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.925542    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf87a86-35e2-449e-8e51-f93ca5c133d3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:31.925619 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.925564    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qcqz\" (UniqueName: \"kubernetes.io/projected/bbf87a86-35e2-449e-8e51-f93ca5c133d3-kube-api-access-6qcqz\") pod \"maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:31.926234 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.926156    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf87a86-35e2-449e-8e51-f93ca5c133d3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:31.928045 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.928024    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf87a86-35e2-449e-8e51-f93ca5c133d3-maas-api-tls\") pod \"maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:31.928152 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.928075    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf87a86-35e2-449e-8e51-f93ca5c133d3-metrics-certs\") pod \"maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:31.939479 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:31.939449    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qcqz\" (UniqueName: \"kubernetes.io/projected/bbf87a86-35e2-449e-8e51-f93ca5c133d3-kube-api-access-6qcqz\") pod \"maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:32.000551 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:32.000518    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:32.077230 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:32.077144    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:32.135452 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:32.135412    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:32.346829 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:32.346807    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"]
Sep 12 00:56:32.348478 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:56:32.348453    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf87a86_35e2_449e_8e51_f93ca5c133d3.slice/crio-1c3e1a224ab9710745aef63b6caee4632c6f4fe3dfa18484153d2598ed7694a2 WatchSource:0}: Error finding container 1c3e1a224ab9710745aef63b6caee4632c6f4fe3dfa18484153d2598ed7694a2: Status 404 returned error can't find the container with id 1c3e1a224ab9710745aef63b6caee4632c6f4fe3dfa18484153d2598ed7694a2
Sep 12 00:56:32.524360 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:32.524330    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:32.550203 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:32.550167    2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b94018a_c6eb_45ce_99ed_e09c01a56abc.slice/crio-a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7\": RecentStats: unable to find data in memory cache]"
Sep 12 00:56:32.603601 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:32.603485    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:32.755499 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:32.755465    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp" event={"ID":"bbf87a86-35e2-449e-8e51-f93ca5c133d3","Type":"ContainerStarted","Data":"1c3e1a224ab9710745aef63b6caee4632c6f4fe3dfa18484153d2598ed7694a2"}
Sep 12 00:56:33.080671 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:33.080625    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:33.761872 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:33.761840    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp" event={"ID":"bbf87a86-35e2-449e-8e51-f93ca5c133d3","Type":"ContainerStarted","Data":"8ad6469033c2779b18554b0f586d923a2516b68ab4a77d27da030fe54a58aab7"}
Sep 12 00:56:33.762104 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:33.761991    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:33.783666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:33.783595    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp" podStartSLOduration=2.330684411 podStartE2EDuration="2.783582132s" totalImagesPullingTime="452.897721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:33.782766244 +0000 UTC m=+798.343217922" watchObservedRunningTime="2026-09-12 00:56:33.783582132 +0000 UTC m=+798.344033810"
Sep 12 00:56:34.409370 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:34.409337    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:35.498683 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:35.498653    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:36.183335 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:36.183309    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:36.982705 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:36.982620    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:37.685383 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:37.685353    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:38.041157 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:38.041083    2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b94018a_c6eb_45ce_99ed_e09c01a56abc.slice/crio-a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7\": RecentStats: unable to find data in memory cache]"
Sep 12 00:56:38.388411 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:38.388323    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:39.206616 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:39.206588    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:39.771986 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:39.771933    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 00:56:39.978511 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:39.978482    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:40.223709 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:40.223673    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:40.300210 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:40.300180    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:40.481646 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:40.481535    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:43.580836 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:43.580801    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:44.729588 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:44.729554    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:45.323301 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:45.323269    2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b94018a_c6eb_45ce_99ed_e09c01a56abc.slice/crio-a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7\": RecentStats: unable to find data in memory cache]"
Sep 12 00:56:45.323466 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:45.323306    2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b94018a_c6eb_45ce_99ed_e09c01a56abc.slice/crio-a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7\": RecentStats: unable to find data in memory cache]"
Sep 12 00:56:45.428905 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.428867    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"]
Sep 12 00:56:45.429144 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.429120    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq" podUID="352b38fc-c8de-451c-a8e2-23b20c33315d" containerName="maas-api" containerID="cri-o://4ad185596154f1722b9b482254834d8456cb8d07df10e3aa51ef97ecd7a9725e" gracePeriod=30
Sep 12 00:56:45.458648 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.458623    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:45.532314 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.532285    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:45.641554 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.641513    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:45.683914 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.683888    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:45.816673 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.816640    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:45.820993 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.820928    2567 generic.go:356] "Generic (PLEG): container finished" podID="352b38fc-c8de-451c-a8e2-23b20c33315d" containerID="4ad185596154f1722b9b482254834d8456cb8d07df10e3aa51ef97ecd7a9725e" exitCode=0
Sep 12 00:56:45.821164 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.820995    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"
Sep 12 00:56:45.821164 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.821044    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq" event={"ID":"352b38fc-c8de-451c-a8e2-23b20c33315d","Type":"ContainerDied","Data":"4ad185596154f1722b9b482254834d8456cb8d07df10e3aa51ef97ecd7a9725e"}
Sep 12 00:56:45.821164 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.821087    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq" event={"ID":"352b38fc-c8de-451c-a8e2-23b20c33315d","Type":"ContainerDied","Data":"3dd876ea1655da75d7bf2a211b4400e68e5ac66c94b7006f4fb165978a93ec6d"}
Sep 12 00:56:45.821164 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.821116    2567 scope.go:117] "RemoveContainer" containerID="4ad185596154f1722b9b482254834d8456cb8d07df10e3aa51ef97ecd7a9725e"
Sep 12 00:56:45.832574 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.832521    2567 scope.go:117] "RemoveContainer" containerID="4ad185596154f1722b9b482254834d8456cb8d07df10e3aa51ef97ecd7a9725e"
Sep 12 00:56:45.832913 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:45.832888    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad185596154f1722b9b482254834d8456cb8d07df10e3aa51ef97ecd7a9725e\": container with ID starting with 4ad185596154f1722b9b482254834d8456cb8d07df10e3aa51ef97ecd7a9725e not found: ID does not exist" containerID="4ad185596154f1722b9b482254834d8456cb8d07df10e3aa51ef97ecd7a9725e"
Sep 12 00:56:45.833015 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.832923    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad185596154f1722b9b482254834d8456cb8d07df10e3aa51ef97ecd7a9725e"} err="failed to get container status \"4ad185596154f1722b9b482254834d8456cb8d07df10e3aa51ef97ecd7a9725e\": rpc error: code = NotFound desc = could not find container \"4ad185596154f1722b9b482254834d8456cb8d07df10e3aa51ef97ecd7a9725e\": container with ID starting with 4ad185596154f1722b9b482254834d8456cb8d07df10e3aa51ef97ecd7a9725e not found: ID does not exist"
Sep 12 00:56:45.859411 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.859338    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352b38fc-c8de-451c-a8e2-23b20c33315d-openshift-service-ca\") pod \"352b38fc-c8de-451c-a8e2-23b20c33315d\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") "
Sep 12 00:56:45.859411 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.859397    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352b38fc-c8de-451c-a8e2-23b20c33315d-metrics-certs\") pod \"352b38fc-c8de-451c-a8e2-23b20c33315d\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") "
Sep 12 00:56:45.859591 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.859449    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352b38fc-c8de-451c-a8e2-23b20c33315d-maas-api-tls\") pod \"352b38fc-c8de-451c-a8e2-23b20c33315d\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") "
Sep 12 00:56:45.859591 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.859517    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjzhf\" (UniqueName: \"kubernetes.io/projected/352b38fc-c8de-451c-a8e2-23b20c33315d-kube-api-access-vjzhf\") pod \"352b38fc-c8de-451c-a8e2-23b20c33315d\" (UID: \"352b38fc-c8de-451c-a8e2-23b20c33315d\") "
Sep 12 00:56:45.859785 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.859755    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/352b38fc-c8de-451c-a8e2-23b20c33315d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "352b38fc-c8de-451c-a8e2-23b20c33315d" (UID: "352b38fc-c8de-451c-a8e2-23b20c33315d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:56:45.861745 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.861715    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/352b38fc-c8de-451c-a8e2-23b20c33315d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "352b38fc-c8de-451c-a8e2-23b20c33315d" (UID: "352b38fc-c8de-451c-a8e2-23b20c33315d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:56:45.861842 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.861803    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352b38fc-c8de-451c-a8e2-23b20c33315d-kube-api-access-vjzhf" (OuterVolumeSpecName: "kube-api-access-vjzhf") pod "352b38fc-c8de-451c-a8e2-23b20c33315d" (UID: "352b38fc-c8de-451c-a8e2-23b20c33315d"). InnerVolumeSpecName "kube-api-access-vjzhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:56:45.861842 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.861808    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/352b38fc-c8de-451c-a8e2-23b20c33315d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "352b38fc-c8de-451c-a8e2-23b20c33315d" (UID: "352b38fc-c8de-451c-a8e2-23b20c33315d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:56:45.912381 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.912343    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:45.960426 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.960391    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352b38fc-c8de-451c-a8e2-23b20c33315d-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:56:45.960426 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.960419    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjzhf\" (UniqueName: \"kubernetes.io/projected/352b38fc-c8de-451c-a8e2-23b20c33315d-kube-api-access-vjzhf\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:56:45.960634 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.960436    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352b38fc-c8de-451c-a8e2-23b20c33315d-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:56:45.960634 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:45.960450    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352b38fc-c8de-451c-a8e2-23b20c33315d-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:56:46.136865 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:46.136787    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"]
Sep 12 00:56:46.141043 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:46.141021    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2dc0e67-754bdf5757-xx2nq"]
Sep 12 00:56:46.209300 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:46.209266    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:46.343585 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:46.343552    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:46.440967 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:46.440916    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:46.572116 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:46.572086    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:46.903425 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:46.903348    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:47.016573 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:47.016541    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:47.282340 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:47.282313    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:47.566985 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:47.566896    2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b94018a_c6eb_45ce_99ed_e09c01a56abc.slice/crio-a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7\": RecentStats: unable to find data in memory cache]"
Sep 12 00:56:47.746292 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:47.746264    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:47.974066 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:47.974036    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="352b38fc-c8de-451c-a8e2-23b20c33315d" path="/var/lib/kubelet/pods/352b38fc-c8de-451c-a8e2-23b20c33315d/volumes"
Sep 12 00:56:48.050590 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:48.050556    2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b94018a_c6eb_45ce_99ed_e09c01a56abc.slice/crio-a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7\": RecentStats: unable to find data in memory cache]"
Sep 12 00:56:48.132486 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:48.132458    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:48.693560 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:48.693531    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:48.807450 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:48.807420    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:48.948531 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:48.948450    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:49.066661 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:49.066620    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:49.166488 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:49.166458    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:49.977509 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:49.977478    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:50.083614 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:50.083581    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:50.180560 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:50.180532    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:51.000637 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:51.000603    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:51.830469 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:51.830433    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:52.686850 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:52.686816    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:53.479549 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.479511    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"]
Sep 12 00:56:53.480036 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.479930    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="352b38fc-c8de-451c-a8e2-23b20c33315d" containerName="maas-api"
Sep 12 00:56:53.480036 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.479941    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="352b38fc-c8de-451c-a8e2-23b20c33315d" containerName="maas-api"
Sep 12 00:56:53.480036 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.480023    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="352b38fc-c8de-451c-a8e2-23b20c33315d" containerName="maas-api"
Sep 12 00:56:53.484835 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.484815    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:53.487605 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.487582    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f22962b3-serving-cert\""
Sep 12 00:56:53.498140 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.498122    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"]
Sep 12 00:56:53.515907 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.515880    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:53.527970 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.527917    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-maas-api-tls\") pod \"maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:53.528125 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.528030    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-openshift-service-ca\") pod \"maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:53.528125 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.528089    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-metrics-certs\") pod \"maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:53.528125 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.528121    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxrbr\" (UniqueName: \"kubernetes.io/projected/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-kube-api-access-vxrbr\") pod \"maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:53.616441 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.616416    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:53.628589 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.628564    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxrbr\" (UniqueName: \"kubernetes.io/projected/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-kube-api-access-vxrbr\") pod \"maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:53.628704 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.628639    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-maas-api-tls\") pod \"maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:53.628704 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.628684    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-openshift-service-ca\") pod \"maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:53.628782 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.628720    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-metrics-certs\") pod \"maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:53.628824 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:53.628790    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-serving-cert: secret "maas-api-e2e-migrate-f22962b3-serving-cert" not found
Sep 12 00:56:53.628887 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:53.628875    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-maas-api-tls podName:1bb51642-a9e3-45d6-9cca-3dfefa60e50f nodeName:}" failed. No retries permitted until 2026-09-12 00:56:54.128854172 +0000 UTC m=+818.689305820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-maas-api-tls") pod "maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj" (UID: "1bb51642-a9e3-45d6-9cca-3dfefa60e50f") : secret "maas-api-e2e-migrate-f22962b3-serving-cert" not found
Sep 12 00:56:53.629398 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.629346    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-openshift-service-ca\") pod \"maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:53.631228 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.631211    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-metrics-certs\") pod \"maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:53.639263 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.639242    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxrbr\" (UniqueName: \"kubernetes.io/projected/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-kube-api-access-vxrbr\") pod \"maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:53.987291 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:53.987259    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:54.087752 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:54.087710    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:54.132618 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:54.132585    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-maas-api-tls\") pod \"maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:54.132768 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:54.132701    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-serving-cert: secret "maas-api-e2e-migrate-f22962b3-serving-cert" not found
Sep 12 00:56:54.132768 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:54.132760    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-maas-api-tls podName:1bb51642-a9e3-45d6-9cca-3dfefa60e50f nodeName:}" failed. No retries permitted until 2026-09-12 00:56:55.13274738 +0000 UTC m=+819.693199026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-maas-api-tls") pod "maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj" (UID: "1bb51642-a9e3-45d6-9cca-3dfefa60e50f") : secret "maas-api-e2e-migrate-f22962b3-serving-cert" not found
Sep 12 00:56:54.239758 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:54.239677    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:54.397831 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:54.397796    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:55.141760 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:55.141732    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-maas-api-tls\") pod \"maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:55.144261 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:55.144243    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-maas-api-tls\") pod \"maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:55.297300 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:55.297268    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:55.399777 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:55.399702    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:55.572864 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:55.572842    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"]
Sep 12 00:56:55.574539 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:56:55.574517    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb51642_a9e3_45d6_9cca_3dfefa60e50f.slice/crio-db26f1a9a698afbe1e7890616e68be522874f70b2d13ce8c36cf40c55b7cdf85 WatchSource:0}: Error finding container db26f1a9a698afbe1e7890616e68be522874f70b2d13ce8c36cf40c55b7cdf85: Status 404 returned error can't find the container with id db26f1a9a698afbe1e7890616e68be522874f70b2d13ce8c36cf40c55b7cdf85
Sep 12 00:56:55.873018 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:55.872988    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj" event={"ID":"1bb51642-a9e3-45d6-9cca-3dfefa60e50f","Type":"ContainerStarted","Data":"db26f1a9a698afbe1e7890616e68be522874f70b2d13ce8c36cf40c55b7cdf85"}
Sep 12 00:56:55.902473 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:55.902441    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:56.001700 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:56.001657    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:56.223661 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:56.223630    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:56.880098 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:56.880015    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj" event={"ID":"1bb51642-a9e3-45d6-9cca-3dfefa60e50f","Type":"ContainerStarted","Data":"f666e5ce374e59e68a1c26d35a40d414d36cdee84249d9e4c2fbac8339a006f6"}
Sep 12 00:56:56.880258 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:56.880157    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:56:56.899373 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:56:56.899328    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj" podStartSLOduration=3.40127301 podStartE2EDuration="3.899312994s" totalImagesPullingTime="498.039984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:56.896556026 +0000 UTC m=+821.457007712" watchObservedRunningTime="2026-09-12 00:56:56.899312994 +0000 UTC m=+821.459764703"
Sep 12 00:56:58.097637 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:56:58.097604    2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b94018a_c6eb_45ce_99ed_e09c01a56abc.slice/crio-a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7\": RecentStats: unable to find data in memory cache]"
Sep 12 00:57:00.223893 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:00.223861    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:02.558246 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:57:02.558208    2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b94018a_c6eb_45ce_99ed_e09c01a56abc.slice/crio-a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7\": RecentStats: unable to find data in memory cache]"
Sep 12 00:57:02.889792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:02.889717    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:57:08.150831 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:57:08.150790    2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b94018a_c6eb_45ce_99ed_e09c01a56abc.slice/crio-a702e7fc5cee810d87fb06e0e58dc8932a782a111d32d3cda0906af32f999ec7\": RecentStats: unable to find data in memory cache]"
Sep 12 00:57:15.491745 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:15.491698    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:16.277666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:16.277585    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:16.958136 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:16.958109    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:17.636963 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:17.636914    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:18.394862 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:18.394823    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:19.099009 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:19.098967    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:19.989560 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:19.989528    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:20.710120 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:20.710085    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:21.505568 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:21.505541    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:22.199632 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:22.199603    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:22.876643 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:22.876610    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:23.685670 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:23.685639    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:24.382168 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:24.382134    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:25.170898 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:25.170870    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:30.530306 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.530274    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"]
Sep 12 00:57:30.534245 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.534229    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.537792 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.537770    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f19fd1\"/\"openshift-service-ca.crt\""
Sep 12 00:57:30.537901 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.537800    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f19fd1\"/\"kube-root-ca.crt\""
Sep 12 00:57:30.537901 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.537776    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f19fd1\"/\"simulator-w4-f19fd1-kserve-self-signed-certs\""
Sep 12 00:57:30.537901 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.537776    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f19fd1\"/\"default-dockercfg-r5zvh\""
Sep 12 00:57:30.545023 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.545001    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"]
Sep 12 00:57:30.635941 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.635909    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:30.660350 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.660321    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-dshm\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.660493 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.660401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-tls-certs\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.660493 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.660434    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-tmp-dir\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.660565 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.660515    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-home\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.660565 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.660548    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-model-cache\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.660634 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.660586    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt652\" (UniqueName: \"kubernetes.io/projected/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-kube-api-access-nt652\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.761764 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.761732    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-tls-certs\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.762004 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.761774    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-tmp-dir\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.762004 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.761814    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-home\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.762004 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.761837    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-model-cache\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.762004 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.761876    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt652\" (UniqueName: \"kubernetes.io/projected/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-kube-api-access-nt652\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.762233 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.762004    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-dshm\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.762296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.762273    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-home\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.762360 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.762338    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-model-cache\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.762417 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.762346    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-tmp-dir\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.764371 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.764350    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-dshm\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.764548 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.764531    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-tls-certs\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.770540 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.770516    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt652\" (UniqueName: \"kubernetes.io/projected/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-kube-api-access-nt652\") pod \"simulator-w4-f19fd1-kserve-69694947cd-t5m7f\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") " pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:30.843245 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:30.843169    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:31.116821 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.116797    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"]
Sep 12 00:57:31.118902 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:57:31.118875    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91095c2f_1f53_4b78_b74e_fd8c1a9e962e.slice/crio-97c35bbe3256bb6b826a8b7d5e46c0ccc6cd5e7070a5b4c87727c8928e52cf17 WatchSource:0}: Error finding container 97c35bbe3256bb6b826a8b7d5e46c0ccc6cd5e7070a5b4c87727c8928e52cf17: Status 404 returned error can't find the container with id 97c35bbe3256bb6b826a8b7d5e46c0ccc6cd5e7070a5b4c87727c8928e52cf17
Sep 12 00:57:31.499596 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.499561    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"]
Sep 12 00:57:31.500056 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.499804    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj" podUID="1bb51642-a9e3-45d6-9cca-3dfefa60e50f" containerName="maas-api" containerID="cri-o://f666e5ce374e59e68a1c26d35a40d414d36cdee84249d9e4c2fbac8339a006f6" gracePeriod=30
Sep 12 00:57:31.527634 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.527593    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:31.634727 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.634681    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:31.727911 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.727884    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:31.731597 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.731576    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:57:31.833505 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.833473    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:31.873884 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.873853    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-maas-api-tls\") pod \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") "
Sep 12 00:57:31.874301 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.874273    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1bb51642-a9e3-45d6-9cca-3dfefa60e50f" (UID: "1bb51642-a9e3-45d6-9cca-3dfefa60e50f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:57:31.874513 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.873927    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-openshift-service-ca\") pod \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") "
Sep 12 00:57:31.874789 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.874771    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-metrics-certs\") pod \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") "
Sep 12 00:57:31.874853 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.874808    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxrbr\" (UniqueName: \"kubernetes.io/projected/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-kube-api-access-vxrbr\") pod \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\" (UID: \"1bb51642-a9e3-45d6-9cca-3dfefa60e50f\") "
Sep 12 00:57:31.875299 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.875276    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:57:31.876447 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.876416    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1bb51642-a9e3-45d6-9cca-3dfefa60e50f" (UID: "1bb51642-a9e3-45d6-9cca-3dfefa60e50f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:57:31.877423 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.877367    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1bb51642-a9e3-45d6-9cca-3dfefa60e50f" (UID: "1bb51642-a9e3-45d6-9cca-3dfefa60e50f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:57:31.878062 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.878025    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-kube-api-access-vxrbr" (OuterVolumeSpecName: "kube-api-access-vxrbr") pod "1bb51642-a9e3-45d6-9cca-3dfefa60e50f" (UID: "1bb51642-a9e3-45d6-9cca-3dfefa60e50f"). InnerVolumeSpecName "kube-api-access-vxrbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:57:31.901755 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.901727    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:31.975835 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.975803    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:57:31.975835 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.975835    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:57:31.976066 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:31.975848    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxrbr\" (UniqueName: \"kubernetes.io/projected/1bb51642-a9e3-45d6-9cca-3dfefa60e50f-kube-api-access-vxrbr\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:57:32.051150 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.051058    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f" event={"ID":"91095c2f-1f53-4b78-b74e-fd8c1a9e962e","Type":"ContainerStarted","Data":"a5484409d6e534867345fdb32bbd0ff981d65b81586e0bb3b36d8b6df2a1a9ef"}
Sep 12 00:57:32.051150 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.051107    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f" event={"ID":"91095c2f-1f53-4b78-b74e-fd8c1a9e962e","Type":"ContainerStarted","Data":"97c35bbe3256bb6b826a8b7d5e46c0ccc6cd5e7070a5b4c87727c8928e52cf17"}
Sep 12 00:57:32.052368 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.052343    2567 generic.go:356] "Generic (PLEG): container finished" podID="1bb51642-a9e3-45d6-9cca-3dfefa60e50f" containerID="f666e5ce374e59e68a1c26d35a40d414d36cdee84249d9e4c2fbac8339a006f6" exitCode=0
Sep 12 00:57:32.052482 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.052380    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"
Sep 12 00:57:32.052482 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.052418    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj" event={"ID":"1bb51642-a9e3-45d6-9cca-3dfefa60e50f","Type":"ContainerDied","Data":"f666e5ce374e59e68a1c26d35a40d414d36cdee84249d9e4c2fbac8339a006f6"}
Sep 12 00:57:32.052482 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.052452    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj" event={"ID":"1bb51642-a9e3-45d6-9cca-3dfefa60e50f","Type":"ContainerDied","Data":"db26f1a9a698afbe1e7890616e68be522874f70b2d13ce8c36cf40c55b7cdf85"}
Sep 12 00:57:32.052482 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.052475    2567 scope.go:117] "RemoveContainer" containerID="f666e5ce374e59e68a1c26d35a40d414d36cdee84249d9e4c2fbac8339a006f6"
Sep 12 00:57:32.063344 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.063328    2567 scope.go:117] "RemoveContainer" containerID="f666e5ce374e59e68a1c26d35a40d414d36cdee84249d9e4c2fbac8339a006f6"
Sep 12 00:57:32.063615 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:57:32.063596    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f666e5ce374e59e68a1c26d35a40d414d36cdee84249d9e4c2fbac8339a006f6\": container with ID starting with f666e5ce374e59e68a1c26d35a40d414d36cdee84249d9e4c2fbac8339a006f6 not found: ID does not exist" containerID="f666e5ce374e59e68a1c26d35a40d414d36cdee84249d9e4c2fbac8339a006f6"
Sep 12 00:57:32.063687 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.063621    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f666e5ce374e59e68a1c26d35a40d414d36cdee84249d9e4c2fbac8339a006f6"} err="failed to get container status \"f666e5ce374e59e68a1c26d35a40d414d36cdee84249d9e4c2fbac8339a006f6\": rpc error: code = NotFound desc = could not find container \"f666e5ce374e59e68a1c26d35a40d414d36cdee84249d9e4c2fbac8339a006f6\": container with ID starting with f666e5ce374e59e68a1c26d35a40d414d36cdee84249d9e4c2fbac8339a006f6 not found: ID does not exist"
Sep 12 00:57:32.070772 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.070732    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f" podStartSLOduration=2.070718981 podStartE2EDuration="2.070718981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:57:32.069542835 +0000 UTC m=+856.629994523" watchObservedRunningTime="2026-09-12 00:57:32.070718981 +0000 UTC m=+856.631170682"
Sep 12 00:57:32.082329 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.082308    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"]
Sep 12 00:57:32.085887 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.085869    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f22962b3-6cdc5b67cc-vbshj"]
Sep 12 00:57:32.194429 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.194406    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:32.328647 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.328575    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:32.646042 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.645931    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:32.975275 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:32.975247    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:33.156751 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:33.156723    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:33.809366 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:33.809339    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:33.973764 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:33.973734    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bb51642-a9e3-45d6-9cca-3dfefa60e50f" path="/var/lib/kubelet/pods/1bb51642-a9e3-45d6-9cca-3dfefa60e50f/volumes"
Sep 12 00:57:34.858612 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:34.858583    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:35.018539 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:35.018510    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:35.116778 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:35.116699    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:35.688243 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:35.688217    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:36.606414 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:36.606370    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:37.303850 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.303818    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"]
Sep 12 00:57:37.304270 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.304254    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bb51642-a9e3-45d6-9cca-3dfefa60e50f" containerName="maas-api"
Sep 12 00:57:37.304270 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.304270    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bb51642-a9e3-45d6-9cca-3dfefa60e50f" containerName="maas-api"
Sep 12 00:57:37.304431 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.304349    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bb51642-a9e3-45d6-9cca-3dfefa60e50f" containerName="maas-api"
Sep 12 00:57:37.309160 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.309138    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:37.311917 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.311893    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6e7edcc4-serving-cert\""
Sep 12 00:57:37.322293 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.322271    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"]
Sep 12 00:57:37.339247 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.339226    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:37.415242 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.415216    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:37.429778 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.429747    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-maas-api-tls\") pod \"maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:37.429906 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.429805    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-openshift-service-ca\") pod \"maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:37.429906 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.429883    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf2vx\" (UniqueName: \"kubernetes.io/projected/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-kube-api-access-tf2vx\") pod \"maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:37.430051 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.430005    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-metrics-certs\") pod \"maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:37.531228 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.531180    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf2vx\" (UniqueName: \"kubernetes.io/projected/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-kube-api-access-tf2vx\") pod \"maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:37.531379 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.531306    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-metrics-certs\") pod \"maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:37.531460 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.531406    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-maas-api-tls\") pod \"maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:37.531460 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.531452    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-openshift-service-ca\") pod \"maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:37.531581 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:57:37.531548    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-serving-cert: secret "maas-api-e2e-ait-6e7edcc4-serving-cert" not found
Sep 12 00:57:37.531638 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:57:37.531624    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-maas-api-tls podName:bdcb0f2e-dd30-4e6c-8f99-f6d554a12566 nodeName:}" failed. No retries permitted until 2026-09-12 00:57:38.031606741 +0000 UTC m=+862.592058396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-maas-api-tls") pod "maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4" (UID: "bdcb0f2e-dd30-4e6c-8f99-f6d554a12566") : secret "maas-api-e2e-ait-6e7edcc4-serving-cert" not found
Sep 12 00:57:37.532157 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.532139    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-openshift-service-ca\") pod \"maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:37.534101 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.534080    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-metrics-certs\") pod \"maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:37.540548 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.540521    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf2vx\" (UniqueName: \"kubernetes.io/projected/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-kube-api-access-tf2vx\") pod \"maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:37.760157 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:37.760110    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:38.037459 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:38.037370    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-maas-api-tls\") pod \"maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:38.039874 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:38.039853    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-maas-api-tls\") pod \"maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:38.061036 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:38.061012    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:38.162265 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:38.162231    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:38.220605 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:38.220575    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:38.499712 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:38.499687    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"]
Sep 12 00:57:38.501815 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:57:38.501785    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdcb0f2e_dd30_4e6c_8f99_f6d554a12566.slice/crio-664f89b118f1e423861f7ab4fac859fee5b2855b8fa9ebf3743d3c30d26ada67 WatchSource:0}: Error finding container 664f89b118f1e423861f7ab4fac859fee5b2855b8fa9ebf3743d3c30d26ada67: Status 404 returned error can't find the container with id 664f89b118f1e423861f7ab4fac859fee5b2855b8fa9ebf3743d3c30d26ada67
Sep 12 00:57:38.592542 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:38.592495    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:39.090239 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:39.090207    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4" event={"ID":"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566","Type":"ContainerStarted","Data":"596cbc59c250444c444ca7c900119f1b8675b29321ee3d3ef935c844dd1ee5e9"}
Sep 12 00:57:39.090609 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:39.090260    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4" event={"ID":"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566","Type":"ContainerStarted","Data":"664f89b118f1e423861f7ab4fac859fee5b2855b8fa9ebf3743d3c30d26ada67"}
Sep 12 00:57:39.090609 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:39.090283    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:39.112675 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:39.112582    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4" podStartSLOduration=1.7583613310000001 podStartE2EDuration="2.112566905s" totalImagesPullingTime="354.205574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:57:39.111389829 +0000 UTC m=+863.671841496" watchObservedRunningTime="2026-09-12 00:57:39.112566905 +0000 UTC m=+863.673018572"
Sep 12 00:57:39.308293 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:39.308256    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:40.053073 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:40.053037    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:40.792285 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:40.792253    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:40.844313 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:40.844280    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:40.844462 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:40.844338    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:40.856843 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:40.856816    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:41.111484 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:41.111408    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 00:57:45.100231 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:45.100203    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:57:59.588672 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:57:59.588636    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:00.255956 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:00.255923    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:01.105729 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:01.105687    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:01.904988 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:01.904938    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:02.597688 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:02.597658    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:03.289524 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:03.289485    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:04.091217 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:04.091175    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:04.800510 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:04.800466    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:05.669206 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:05.669164    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:06.338047 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:06.338018    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:07.011397 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.011306    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"]
Sep 12 00:58:07.011855 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.011578    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4" podUID="bdcb0f2e-dd30-4e6c-8f99-f6d554a12566" containerName="maas-api" containerID="cri-o://596cbc59c250444c444ca7c900119f1b8675b29321ee3d3ef935c844dd1ee5e9" gracePeriod=30
Sep 12 00:58:07.044242 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.044218    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:07.119168 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.119128    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:07.180940 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.180880    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:07.235407 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.235380    2567 generic.go:356] "Generic (PLEG): container finished" podID="bdcb0f2e-dd30-4e6c-8f99-f6d554a12566" containerID="596cbc59c250444c444ca7c900119f1b8675b29321ee3d3ef935c844dd1ee5e9" exitCode=0
Sep 12 00:58:07.235573 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.235465    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4" event={"ID":"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566","Type":"ContainerDied","Data":"596cbc59c250444c444ca7c900119f1b8675b29321ee3d3ef935c844dd1ee5e9"}
Sep 12 00:58:07.256153 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.256123    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:07.266401 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.266347    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:58:07.343005 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.342978    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:07.415176 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.415151    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-metrics-certs\") pod \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") "
Sep 12 00:58:07.415345 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.415196    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf2vx\" (UniqueName: \"kubernetes.io/projected/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-kube-api-access-tf2vx\") pod \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") "
Sep 12 00:58:07.415345 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.415247    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-maas-api-tls\") pod \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") "
Sep 12 00:58:07.415345 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.415341    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-openshift-service-ca\") pod \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\" (UID: \"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566\") "
Sep 12 00:58:07.415761 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.415728    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bdcb0f2e-dd30-4e6c-8f99-f6d554a12566" (UID: "bdcb0f2e-dd30-4e6c-8f99-f6d554a12566"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:58:07.417455 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.417431    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bdcb0f2e-dd30-4e6c-8f99-f6d554a12566" (UID: "bdcb0f2e-dd30-4e6c-8f99-f6d554a12566"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:58:07.417545 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.417454    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bdcb0f2e-dd30-4e6c-8f99-f6d554a12566" (UID: "bdcb0f2e-dd30-4e6c-8f99-f6d554a12566"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:58:07.417545 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.417486    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-kube-api-access-tf2vx" (OuterVolumeSpecName: "kube-api-access-tf2vx") pod "bdcb0f2e-dd30-4e6c-8f99-f6d554a12566" (UID: "bdcb0f2e-dd30-4e6c-8f99-f6d554a12566"). InnerVolumeSpecName "kube-api-access-tf2vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:58:07.516934 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.516864    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:07.516934 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.516890    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:07.516934 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.516899    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf2vx\" (UniqueName: \"kubernetes.io/projected/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-kube-api-access-tf2vx\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:07.516934 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.516907    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:07.765364 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:07.765334    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:08.241507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:08.241418    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"
Sep 12 00:58:08.241507 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:08.241444    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4" event={"ID":"bdcb0f2e-dd30-4e6c-8f99-f6d554a12566","Type":"ContainerDied","Data":"664f89b118f1e423861f7ab4fac859fee5b2855b8fa9ebf3743d3c30d26ada67"}
Sep 12 00:58:08.241982 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:08.241512    2567 scope.go:117] "RemoveContainer" containerID="596cbc59c250444c444ca7c900119f1b8675b29321ee3d3ef935c844dd1ee5e9"
Sep 12 00:58:08.282309 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:08.282281    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"]
Sep 12 00:58:08.291891 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:08.291868    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e7edcc4-8695dd7c75-zvxb4"]
Sep 12 00:58:08.482735 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:08.482701    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:08.694363 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:08.694333    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:08.915676 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:08.915646    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:09.232097 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:09.232062    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:09.586845 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:09.586761    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:09.928958 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:09.928917    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:09.973538 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:09.973511    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdcb0f2e-dd30-4e6c-8f99-f6d554a12566" path="/var/lib/kubelet/pods/bdcb0f2e-dd30-4e6c-8f99-f6d554a12566/volumes"
Sep 12 00:58:10.309210 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:10.309134    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:11.089822 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:11.089789    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:11.255200 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:11.255152    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:11.355092 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:11.355015    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:12.039384 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.039348    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:12.775444 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.775413    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"]
Sep 12 00:58:12.776036 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.776020    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdcb0f2e-dd30-4e6c-8f99-f6d554a12566" containerName="maas-api"
Sep 12 00:58:12.776092 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.776039    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdcb0f2e-dd30-4e6c-8f99-f6d554a12566" containerName="maas-api"
Sep 12 00:58:12.776148 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.776137    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdcb0f2e-dd30-4e6c-8f99-f6d554a12566" containerName="maas-api"
Sep 12 00:58:12.779673 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.779651    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:12.782303 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.782280    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-7c4848b5-serving-cert\""
Sep 12 00:58:12.790286 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.790265    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"]
Sep 12 00:58:12.808653 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.808629    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:12.864987 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.864931    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd51dd9-84a7-4305-8ca8-715e50f462d0-openshift-service-ca\") pod \"maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:12.865114 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.865013    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd51dd9-84a7-4305-8ca8-715e50f462d0-metrics-certs\") pod \"maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:12.865114 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.865040    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9bbr\" (UniqueName: \"kubernetes.io/projected/9cd51dd9-84a7-4305-8ca8-715e50f462d0-kube-api-access-h9bbr\") pod \"maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:12.865211 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.865176    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd51dd9-84a7-4305-8ca8-715e50f462d0-maas-api-tls\") pod \"maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:12.881210 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.881190    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:12.948533 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.948508    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"]
Sep 12 00:58:12.952041 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.952020    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:12.954734 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.954715    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-64f698\"/\"autores-model-64f698-kserve-self-signed-certs\""
Sep 12 00:58:12.954734 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.954727    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-64f698\"/\"default-dockercfg-st7vh\""
Sep 12 00:58:12.954929 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.954746    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-64f698\"/\"kube-root-ca.crt\""
Sep 12 00:58:12.954929 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.954766    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-64f698\"/\"openshift-service-ca.crt\""
Sep 12 00:58:12.964269 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.964246    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"]
Sep 12 00:58:12.966478 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.966453    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd51dd9-84a7-4305-8ca8-715e50f462d0-maas-api-tls\") pod \"maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:12.966571 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.966540    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd51dd9-84a7-4305-8ca8-715e50f462d0-openshift-service-ca\") pod \"maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:12.966650 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.966581    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd51dd9-84a7-4305-8ca8-715e50f462d0-metrics-certs\") pod \"maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:12.966650 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.966597    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9bbr\" (UniqueName: \"kubernetes.io/projected/9cd51dd9-84a7-4305-8ca8-715e50f462d0-kube-api-access-h9bbr\") pod \"maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:12.967554 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.967530    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd51dd9-84a7-4305-8ca8-715e50f462d0-openshift-service-ca\") pod \"maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:12.969209 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.969184    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd51dd9-84a7-4305-8ca8-715e50f462d0-metrics-certs\") pod \"maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:12.969209 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.969202    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd51dd9-84a7-4305-8ca8-715e50f462d0-maas-api-tls\") pod \"maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:12.974764 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.974742    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:12.977736 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:12.977713    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9bbr\" (UniqueName: \"kubernetes.io/projected/9cd51dd9-84a7-4305-8ca8-715e50f462d0-kube-api-access-h9bbr\") pod \"maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:13.061484 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.061410    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:13.067549 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.067530    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-tmp-dir\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.067663 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.067558    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d64b915-3be3-42e9-909e-720fe60f7b5a-tls-certs\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.067710 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.067655    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-dshm\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.067710 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.067699    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-model-cache\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.067790 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.067718    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltmct\" (UniqueName: \"kubernetes.io/projected/7d64b915-3be3-42e9-909e-720fe60f7b5a-kube-api-access-ltmct\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.067790 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.067768    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-home\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.092930 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.092909    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:13.168406 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.168376    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-tmp-dir\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.168530 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.168419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d64b915-3be3-42e9-909e-720fe60f7b5a-tls-certs\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.168530 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.168508    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-dshm\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.168597 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.168555    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-model-cache\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.168597 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.168584    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmct\" (UniqueName: \"kubernetes.io/projected/7d64b915-3be3-42e9-909e-720fe60f7b5a-kube-api-access-ltmct\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.168822 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.168797    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-tmp-dir\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.168965 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.168894    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-home\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.169061 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.168989    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-model-cache\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.169219 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.169198    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-home\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.171442 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.171418    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-dshm\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.172029 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.172005    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d64b915-3be3-42e9-909e-720fe60f7b5a-tls-certs\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.179751 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.179728    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmct\" (UniqueName: \"kubernetes.io/projected/7d64b915-3be3-42e9-909e-720fe60f7b5a-kube-api-access-ltmct\") pod \"autores-model-64f698-kserve-78594f8f5f-8vwlz\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") " pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.261425 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.261396    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:13.371140 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.371108    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:13.371774 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.371749    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"]
Sep 12 00:58:13.374783 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:58:13.374759    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd51dd9_84a7_4305_8ca8_715e50f462d0.slice/crio-12b7f93e43b859a17954a7b6ac34a822b82f7bbbfeec27d8dedb429b52081586 WatchSource:0}: Error finding container 12b7f93e43b859a17954a7b6ac34a822b82f7bbbfeec27d8dedb429b52081586: Status 404 returned error can't find the container with id 12b7f93e43b859a17954a7b6ac34a822b82f7bbbfeec27d8dedb429b52081586
Sep 12 00:58:13.469041 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.469005    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:13.545720 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.545686    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"]
Sep 12 00:58:13.547867 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:58:13.547825    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d64b915_3be3_42e9_909e_720fe60f7b5a.slice/crio-1a696b1dc051707c9d5ae186e37f6fceb72341e7436a6f7717f7018b50fbf18f WatchSource:0}: Error finding container 1a696b1dc051707c9d5ae186e37f6fceb72341e7436a6f7717f7018b50fbf18f: Status 404 returned error can't find the container with id 1a696b1dc051707c9d5ae186e37f6fceb72341e7436a6f7717f7018b50fbf18f
Sep 12 00:58:13.800763 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:13.800735    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:14.275557 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:14.275515    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8" event={"ID":"9cd51dd9-84a7-4305-8ca8-715e50f462d0","Type":"ContainerStarted","Data":"69b5a6dfbb729243ad629f9151894f111713480a2d0a1ffc904233c0a4bb0fa2"}
Sep 12 00:58:14.275728 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:14.275587    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:14.275728 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:14.275610    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8" event={"ID":"9cd51dd9-84a7-4305-8ca8-715e50f462d0","Type":"ContainerStarted","Data":"12b7f93e43b859a17954a7b6ac34a822b82f7bbbfeec27d8dedb429b52081586"}
Sep 12 00:58:14.277200 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:14.277170    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz" event={"ID":"7d64b915-3be3-42e9-909e-720fe60f7b5a","Type":"ContainerStarted","Data":"2cb4e178ac249913fecb782f4dc79a2a0fa3c9680311e00a64f3e7e194f066ba"}
Sep 12 00:58:14.277320 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:14.277206    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz" event={"ID":"7d64b915-3be3-42e9-909e-720fe60f7b5a","Type":"ContainerStarted","Data":"1a696b1dc051707c9d5ae186e37f6fceb72341e7436a6f7717f7018b50fbf18f"}
Sep 12 00:58:14.295490 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:14.295449    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8" podStartSLOduration=1.871174362 podStartE2EDuration="2.295435305s" totalImagesPullingTime="424.260943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:58:14.293401205 +0000 UTC m=+898.853852906" watchObservedRunningTime="2026-09-12 00:58:14.295435305 +0000 UTC m=+898.855886973"
Sep 12 00:58:14.310698 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:14.310655    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz" podStartSLOduration=2.310641246 podStartE2EDuration="2.310641246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:58:14.309429396 +0000 UTC m=+898.869881064" watchObservedRunningTime="2026-09-12 00:58:14.310641246 +0000 UTC m=+898.871092913"
Sep 12 00:58:14.522603 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:14.522570    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:15.244038 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:15.244003    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:15.901356 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:15.901320    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 00:58:15.902649 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:15.902625    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 00:58:15.905319 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:15.905293    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:16.699211 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:16.699175    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:19.219284 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:19.219250    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:20.289357 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:20.289328    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:23.262235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:23.262204    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:23.262235 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:23.262242    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:23.275846 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:23.275822    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:23.335124 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:23.335096    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:58:33.881450 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:33.881416    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:35.136140 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:35.136100    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:35.358934 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:35.358898    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:35.959768 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:35.959735    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:36.804391 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:36.804356    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:37.489691 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:37.489660    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:38.224071 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:38.224038    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:38.899328 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:38.899288    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:39.711250 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:39.711222    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:40.375812 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:40.375778    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:41.110925 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:41.110889    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:42.693028 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:42.692997    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:43.285843 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.285812    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"]
Sep 12 00:58:43.286072 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.286035    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8" podUID="9cd51dd9-84a7-4305-8ca8-715e50f462d0" containerName="maas-api" containerID="cri-o://69b5a6dfbb729243ad629f9151894f111713480a2d0a1ffc904233c0a4bb0fa2" gracePeriod=30
Sep 12 00:58:43.312242 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.312217    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:43.382869 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.382792    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:43.427473 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.427447    2567 generic.go:356] "Generic (PLEG): container finished" podID="9cd51dd9-84a7-4305-8ca8-715e50f462d0" containerID="69b5a6dfbb729243ad629f9151894f111713480a2d0a1ffc904233c0a4bb0fa2" exitCode=0
Sep 12 00:58:43.427629 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.427529    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8" event={"ID":"9cd51dd9-84a7-4305-8ca8-715e50f462d0","Type":"ContainerDied","Data":"69b5a6dfbb729243ad629f9151894f111713480a2d0a1ffc904233c0a4bb0fa2"}
Sep 12 00:58:43.447917 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.447889    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:43.516536 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.516505    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:43.544339 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.544284    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:43.645373 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.645335    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd51dd9-84a7-4305-8ca8-715e50f462d0-metrics-certs\") pod \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") "
Sep 12 00:58:43.645572 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.645464    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd51dd9-84a7-4305-8ca8-715e50f462d0-maas-api-tls\") pod \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") "
Sep 12 00:58:43.645572 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.645560    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9bbr\" (UniqueName: \"kubernetes.io/projected/9cd51dd9-84a7-4305-8ca8-715e50f462d0-kube-api-access-h9bbr\") pod \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") "
Sep 12 00:58:43.647231 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.645607    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd51dd9-84a7-4305-8ca8-715e50f462d0-openshift-service-ca\") pod \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\" (UID: \"9cd51dd9-84a7-4305-8ca8-715e50f462d0\") "
Sep 12 00:58:43.647231 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.646461    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd51dd9-84a7-4305-8ca8-715e50f462d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9cd51dd9-84a7-4305-8ca8-715e50f462d0" (UID: "9cd51dd9-84a7-4305-8ca8-715e50f462d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:58:43.648296 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.648130    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd51dd9-84a7-4305-8ca8-715e50f462d0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9cd51dd9-84a7-4305-8ca8-715e50f462d0" (UID: "9cd51dd9-84a7-4305-8ca8-715e50f462d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:58:43.648423 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.648368    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd51dd9-84a7-4305-8ca8-715e50f462d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9cd51dd9-84a7-4305-8ca8-715e50f462d0" (UID: "9cd51dd9-84a7-4305-8ca8-715e50f462d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:58:43.651376 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.651348    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd51dd9-84a7-4305-8ca8-715e50f462d0-kube-api-access-h9bbr" (OuterVolumeSpecName: "kube-api-access-h9bbr") pod "9cd51dd9-84a7-4305-8ca8-715e50f462d0" (UID: "9cd51dd9-84a7-4305-8ca8-715e50f462d0"). InnerVolumeSpecName "kube-api-access-h9bbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:58:43.676044 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.676014    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:43.746800 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.746764    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd51dd9-84a7-4305-8ca8-715e50f462d0-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:43.746800 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.746787    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd51dd9-84a7-4305-8ca8-715e50f462d0-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:43.746800 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.746796    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9bbr\" (UniqueName: \"kubernetes.io/projected/9cd51dd9-84a7-4305-8ca8-715e50f462d0-kube-api-access-h9bbr\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:43.746800 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:43.746807    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd51dd9-84a7-4305-8ca8-715e50f462d0-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:44.004439 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:44.004411    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:44.347430 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:44.347334    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:44.433967 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:44.433925    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8" event={"ID":"9cd51dd9-84a7-4305-8ca8-715e50f462d0","Type":"ContainerDied","Data":"12b7f93e43b859a17954a7b6ac34a822b82f7bbbfeec27d8dedb429b52081586"}
Sep 12 00:58:44.434161 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:44.434013    2567 scope.go:117] "RemoveContainer" containerID="69b5a6dfbb729243ad629f9151894f111713480a2d0a1ffc904233c0a4bb0fa2"
Sep 12 00:58:44.434161 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:44.433934    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"
Sep 12 00:58:44.441868 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:44.441836    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:44.450530 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:44.450503    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"]
Sep 12 00:58:44.455089 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:44.455067    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7c4848b5-5dd6fbdf96-rgdl8"]
Sep 12 00:58:44.544478 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:44.544445    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:44.910670 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:44.910635    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:45.791734 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:45.791701    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:45.916138 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:45.916107    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:45.974050 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:45.974022    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd51dd9-84a7-4305-8ca8-715e50f462d0" path="/var/lib/kubelet/pods/9cd51dd9-84a7-4305-8ca8-715e50f462d0/volumes"
Sep 12 00:58:46.516299 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:46.516267    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:47.525834 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:47.525803    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:48.515309 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:48.515275    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:49.421941 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:49.421877    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:50.306107 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:50.306071    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:50.991600 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:50.991567    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:51.800620 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:51.800591    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:52.042117 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:52.042087    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:52.151667 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:52.151585    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:52.244118 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:52.244086    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:52.344998 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:52.344943    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:52.777769 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:52.777730    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:57.106301 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.106271    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"]
Sep 12 00:58:57.107001 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.106535    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9" podUID="72b1b072-5cd0-4677-b516-8cd0b234ad2e" containerName="maas-api" containerID="cri-o://bc7946b662bf8982a0df0f2a85e3318aa1a14dca6e60830a7ab2ed79fe78529a" gracePeriod=30
Sep 12 00:58:57.146042 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.145992    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:57.224761 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.224733    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:57.294820 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.294786    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:57.352503 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.352482    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:58:57.387642 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.387569    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:57.456614 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.456585    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:57.491808 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.491779    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72b1b072-5cd0-4677-b516-8cd0b234ad2e-metrics-certs\") pod \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") "
Sep 12 00:58:57.491924 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.491878    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72b1b072-5cd0-4677-b516-8cd0b234ad2e-maas-api-tls\") pod \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") "
Sep 12 00:58:57.491924 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.491898    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72b1b072-5cd0-4677-b516-8cd0b234ad2e-openshift-service-ca\") pod \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") "
Sep 12 00:58:57.492040 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.491934    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zq8j\" (UniqueName: \"kubernetes.io/projected/72b1b072-5cd0-4677-b516-8cd0b234ad2e-kube-api-access-5zq8j\") pod \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\" (UID: \"72b1b072-5cd0-4677-b516-8cd0b234ad2e\") "
Sep 12 00:58:57.492335 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.492309    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b1b072-5cd0-4677-b516-8cd0b234ad2e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "72b1b072-5cd0-4677-b516-8cd0b234ad2e" (UID: "72b1b072-5cd0-4677-b516-8cd0b234ad2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:58:57.494233 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.494208    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b1b072-5cd0-4677-b516-8cd0b234ad2e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "72b1b072-5cd0-4677-b516-8cd0b234ad2e" (UID: "72b1b072-5cd0-4677-b516-8cd0b234ad2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:58:57.494386 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.494327    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b1b072-5cd0-4677-b516-8cd0b234ad2e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "72b1b072-5cd0-4677-b516-8cd0b234ad2e" (UID: "72b1b072-5cd0-4677-b516-8cd0b234ad2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:58:57.494519 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.494442    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b1b072-5cd0-4677-b516-8cd0b234ad2e-kube-api-access-5zq8j" (OuterVolumeSpecName: "kube-api-access-5zq8j") pod "72b1b072-5cd0-4677-b516-8cd0b234ad2e" (UID: "72b1b072-5cd0-4677-b516-8cd0b234ad2e"). InnerVolumeSpecName "kube-api-access-5zq8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:58:57.500780 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.500754    2567 generic.go:356] "Generic (PLEG): container finished" podID="72b1b072-5cd0-4677-b516-8cd0b234ad2e" containerID="bc7946b662bf8982a0df0f2a85e3318aa1a14dca6e60830a7ab2ed79fe78529a" exitCode=0
Sep 12 00:58:57.500869 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.500841    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9" event={"ID":"72b1b072-5cd0-4677-b516-8cd0b234ad2e","Type":"ContainerDied","Data":"bc7946b662bf8982a0df0f2a85e3318aa1a14dca6e60830a7ab2ed79fe78529a"}
Sep 12 00:58:57.500869 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.500852    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"
Sep 12 00:58:57.500972 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.500897    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9" event={"ID":"72b1b072-5cd0-4677-b516-8cd0b234ad2e","Type":"ContainerDied","Data":"bea8597d08adcb83539e662e05e5b064870d5ff86e25ad8f403e3cadb80291af"}
Sep 12 00:58:57.500972 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.500929    2567 scope.go:117] "RemoveContainer" containerID="bc7946b662bf8982a0df0f2a85e3318aa1a14dca6e60830a7ab2ed79fe78529a"
Sep 12 00:58:57.516787 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.516770    2567 scope.go:117] "RemoveContainer" containerID="bc7946b662bf8982a0df0f2a85e3318aa1a14dca6e60830a7ab2ed79fe78529a"
Sep 12 00:58:57.517073 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:58:57.517053    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7946b662bf8982a0df0f2a85e3318aa1a14dca6e60830a7ab2ed79fe78529a\": container with ID starting with bc7946b662bf8982a0df0f2a85e3318aa1a14dca6e60830a7ab2ed79fe78529a not found: ID does not exist" containerID="bc7946b662bf8982a0df0f2a85e3318aa1a14dca6e60830a7ab2ed79fe78529a"
Sep 12 00:58:57.517139 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.517081    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7946b662bf8982a0df0f2a85e3318aa1a14dca6e60830a7ab2ed79fe78529a"} err="failed to get container status \"bc7946b662bf8982a0df0f2a85e3318aa1a14dca6e60830a7ab2ed79fe78529a\": rpc error: code = NotFound desc = could not find container \"bc7946b662bf8982a0df0f2a85e3318aa1a14dca6e60830a7ab2ed79fe78529a\": container with ID starting with bc7946b662bf8982a0df0f2a85e3318aa1a14dca6e60830a7ab2ed79fe78529a not found: ID does not exist"
Sep 12 00:58:57.532050 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.532029    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"]
Sep 12 00:58:57.536849 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.536829    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-64f698-68f48b5cb6-l8nq9"]
Sep 12 00:58:57.593731 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.593710    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zq8j\" (UniqueName: \"kubernetes.io/projected/72b1b072-5cd0-4677-b516-8cd0b234ad2e-kube-api-access-5zq8j\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:57.593818 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.593734    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72b1b072-5cd0-4677-b516-8cd0b234ad2e-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:57.593818 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.593747    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72b1b072-5cd0-4677-b516-8cd0b234ad2e-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:57.593818 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.593755    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72b1b072-5cd0-4677-b516-8cd0b234ad2e-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:57.740209 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.740185    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:57.812648 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.812603    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:57.973778 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:57.973741    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72b1b072-5cd0-4677-b516-8cd0b234ad2e" path="/var/lib/kubelet/pods/72b1b072-5cd0-4677-b516-8cd0b234ad2e/volumes"
Sep 12 00:58:58.665523 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:58.665444    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:59.393105 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:58:59.393069    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:00.224658 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:00.224616    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:01.091576 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:01.091531    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:02.721854 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:02.721821    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"]
Sep 12 00:59:02.816603 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:02.816515    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:10.137083 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:10.137049    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:17.741191 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:17.741134    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz" podUID="7d64b915-3be3-42e9-909e-720fe60f7b5a" containerName="main" containerID="cri-o://2cb4e178ac249913fecb782f4dc79a2a0fa3c9680311e00a64f3e7e194f066ba" gracePeriod=45
Sep 12 00:59:17.999911 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:17.999851    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:59:18.081814 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.081781    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d64b915-3be3-42e9-909e-720fe60f7b5a-tls-certs\") pod \"7d64b915-3be3-42e9-909e-720fe60f7b5a\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") "
Sep 12 00:59:18.081814 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.081819    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-tmp-dir\") pod \"7d64b915-3be3-42e9-909e-720fe60f7b5a\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") "
Sep 12 00:59:18.082083 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.081840    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-home\") pod \"7d64b915-3be3-42e9-909e-720fe60f7b5a\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") "
Sep 12 00:59:18.082083 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.081862    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-dshm\") pod \"7d64b915-3be3-42e9-909e-720fe60f7b5a\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") "
Sep 12 00:59:18.082083 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.081908    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltmct\" (UniqueName: \"kubernetes.io/projected/7d64b915-3be3-42e9-909e-720fe60f7b5a-kube-api-access-ltmct\") pod \"7d64b915-3be3-42e9-909e-720fe60f7b5a\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") "
Sep 12 00:59:18.082083 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.082039    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-model-cache\") pod \"7d64b915-3be3-42e9-909e-720fe60f7b5a\" (UID: \"7d64b915-3be3-42e9-909e-720fe60f7b5a\") "
Sep 12 00:59:18.082289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.082125    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d64b915-3be3-42e9-909e-720fe60f7b5a" (UID: "7d64b915-3be3-42e9-909e-720fe60f7b5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:18.082289 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.082138    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-home" (OuterVolumeSpecName: "home") pod "7d64b915-3be3-42e9-909e-720fe60f7b5a" (UID: "7d64b915-3be3-42e9-909e-720fe60f7b5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:18.082407 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.082289    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d64b915-3be3-42e9-909e-720fe60f7b5a" (UID: "7d64b915-3be3-42e9-909e-720fe60f7b5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:18.082407 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.082369    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-tmp-dir\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:59:18.082407 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.082383    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-home\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:59:18.082407 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.082393    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-model-cache\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:59:18.084305 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.084280    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-dshm" (OuterVolumeSpecName: "dshm") pod "7d64b915-3be3-42e9-909e-720fe60f7b5a" (UID: "7d64b915-3be3-42e9-909e-720fe60f7b5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:18.084406 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.084331    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d64b915-3be3-42e9-909e-720fe60f7b5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d64b915-3be3-42e9-909e-720fe60f7b5a" (UID: "7d64b915-3be3-42e9-909e-720fe60f7b5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:59:18.084406 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.084373    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d64b915-3be3-42e9-909e-720fe60f7b5a-kube-api-access-ltmct" (OuterVolumeSpecName: "kube-api-access-ltmct") pod "7d64b915-3be3-42e9-909e-720fe60f7b5a" (UID: "7d64b915-3be3-42e9-909e-720fe60f7b5a"). InnerVolumeSpecName "kube-api-access-ltmct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:59:18.183092 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.183045    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d64b915-3be3-42e9-909e-720fe60f7b5a-tls-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:59:18.183092 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.183086    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d64b915-3be3-42e9-909e-720fe60f7b5a-dshm\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:59:18.183092 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.183096    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltmct\" (UniqueName: \"kubernetes.io/projected/7d64b915-3be3-42e9-909e-720fe60f7b5a-kube-api-access-ltmct\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 00:59:18.603062 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.603032    2567 generic.go:356] "Generic (PLEG): container finished" podID="7d64b915-3be3-42e9-909e-720fe60f7b5a" containerID="2cb4e178ac249913fecb782f4dc79a2a0fa3c9680311e00a64f3e7e194f066ba" exitCode=0
Sep 12 00:59:18.603258 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.603072    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz" event={"ID":"7d64b915-3be3-42e9-909e-720fe60f7b5a","Type":"ContainerDied","Data":"2cb4e178ac249913fecb782f4dc79a2a0fa3c9680311e00a64f3e7e194f066ba"}
Sep 12 00:59:18.603258 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.603073    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"
Sep 12 00:59:18.603258 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.603096    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz" event={"ID":"7d64b915-3be3-42e9-909e-720fe60f7b5a","Type":"ContainerDied","Data":"1a696b1dc051707c9d5ae186e37f6fceb72341e7436a6f7717f7018b50fbf18f"}
Sep 12 00:59:18.603258 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.603122    2567 scope.go:117] "RemoveContainer" containerID="2cb4e178ac249913fecb782f4dc79a2a0fa3c9680311e00a64f3e7e194f066ba"
Sep 12 00:59:18.612399 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.612381    2567 scope.go:117] "RemoveContainer" containerID="2cb4e178ac249913fecb782f4dc79a2a0fa3c9680311e00a64f3e7e194f066ba"
Sep 12 00:59:18.612659 ip-10-0-139-121 kubenswrapper[2567]: E0912 00:59:18.612640    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb4e178ac249913fecb782f4dc79a2a0fa3c9680311e00a64f3e7e194f066ba\": container with ID starting with 2cb4e178ac249913fecb782f4dc79a2a0fa3c9680311e00a64f3e7e194f066ba not found: ID does not exist" containerID="2cb4e178ac249913fecb782f4dc79a2a0fa3c9680311e00a64f3e7e194f066ba"
Sep 12 00:59:18.612698 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.612668    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb4e178ac249913fecb782f4dc79a2a0fa3c9680311e00a64f3e7e194f066ba"} err="failed to get container status \"2cb4e178ac249913fecb782f4dc79a2a0fa3c9680311e00a64f3e7e194f066ba\": rpc error: code = NotFound desc = could not find container \"2cb4e178ac249913fecb782f4dc79a2a0fa3c9680311e00a64f3e7e194f066ba\": container with ID starting with 2cb4e178ac249913fecb782f4dc79a2a0fa3c9680311e00a64f3e7e194f066ba not found: ID does not exist"
Sep 12 00:59:18.623585 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.623564    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"]
Sep 12 00:59:18.632084 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.630904    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-64f698/autores-model-64f698-kserve-78594f8f5f-8vwlz"]
Sep 12 00:59:18.713268 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:18.713237    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:19.974502 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:19.974467    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d64b915-3be3-42e9-909e-720fe60f7b5a" path="/var/lib/kubelet/pods/7d64b915-3be3-42e9-909e-720fe60f7b5a/volumes"
Sep 12 00:59:24.598978 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:24.598921    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:24.702567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:24.702526    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:24.798666 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:24.798635    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:27.342214 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.342180    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"]
Sep 12 00:59:27.342604 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.342585    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd51dd9-84a7-4305-8ca8-715e50f462d0" containerName="maas-api"
Sep 12 00:59:27.342604 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.342597    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd51dd9-84a7-4305-8ca8-715e50f462d0" containerName="maas-api"
Sep 12 00:59:27.342687 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.342617    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d64b915-3be3-42e9-909e-720fe60f7b5a" containerName="main"
Sep 12 00:59:27.342687 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.342623    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d64b915-3be3-42e9-909e-720fe60f7b5a" containerName="main"
Sep 12 00:59:27.342687 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.342629    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b1b072-5cd0-4677-b516-8cd0b234ad2e" containerName="maas-api"
Sep 12 00:59:27.342687 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.342634    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b1b072-5cd0-4677-b516-8cd0b234ad2e" containerName="maas-api"
Sep 12 00:59:27.342822 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.342690    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd51dd9-84a7-4305-8ca8-715e50f462d0" containerName="maas-api"
Sep 12 00:59:27.342822 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.342698    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b1b072-5cd0-4677-b516-8cd0b234ad2e" containerName="maas-api"
Sep 12 00:59:27.342822 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.342707    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d64b915-3be3-42e9-909e-720fe60f7b5a" containerName="main"
Sep 12 00:59:27.347091 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.347074    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.350840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.350804    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-10ba6f\"/\"openshift-service-ca.crt\""
Sep 12 00:59:27.350840 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.350823    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10ba6f\"/\"premium-w0-10ba6f-kserve-self-signed-certs\""
Sep 12 00:59:27.351032 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.350850    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10ba6f\"/\"default-dockercfg-fk6m9\""
Sep 12 00:59:27.351032 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.350927    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-10ba6f\"/\"kube-root-ca.crt\""
Sep 12 00:59:27.362482 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.362460    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"]
Sep 12 00:59:27.379922 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.379873    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:27.475467 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.475429    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-dshm\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.475689 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.475483    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/682f03ec-d30f-4129-89f9-9259b9c8bc77-tls-certs\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.475689 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.475566    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-model-cache\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.475689 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.475656    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-home\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.475800 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.475703    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p28vh\" (UniqueName: \"kubernetes.io/projected/682f03ec-d30f-4129-89f9-9259b9c8bc77-kube-api-access-p28vh\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.475800 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.475761    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-tmp-dir\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.577193 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.577156    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/682f03ec-d30f-4129-89f9-9259b9c8bc77-tls-certs\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.577343 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.577215    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-model-cache\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.577343 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.577251    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-home\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.577343 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.577276    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p28vh\" (UniqueName: \"kubernetes.io/projected/682f03ec-d30f-4129-89f9-9259b9c8bc77-kube-api-access-p28vh\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.577343 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.577296    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-tmp-dir\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.577343 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.577328    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-dshm\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.577740 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.577699    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-model-cache\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.577859 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.577749    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-tmp-dir\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.577859 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.577785    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-home\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.579743 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.579723    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-dshm\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.579970 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.579919    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/682f03ec-d30f-4129-89f9-9259b9c8bc77-tls-certs\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.585446 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.585426    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p28vh\" (UniqueName: \"kubernetes.io/projected/682f03ec-d30f-4129-89f9-9259b9c8bc77-kube-api-access-p28vh\") pod \"premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") " pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.655904 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.655823    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:27.927349 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:27.927289    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"]
Sep 12 00:59:27.929716 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:59:27.929682    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod682f03ec_d30f_4129_89f9_9259b9c8bc77.slice/crio-5d67d6423616860c23703880f7171dc5126885a54b7ee064a1115c90a7d1fe8d WatchSource:0}: Error finding container 5d67d6423616860c23703880f7171dc5126885a54b7ee064a1115c90a7d1fe8d: Status 404 returned error can't find the container with id 5d67d6423616860c23703880f7171dc5126885a54b7ee064a1115c90a7d1fe8d
Sep 12 00:59:28.652838 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:28.652803    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq" event={"ID":"682f03ec-d30f-4129-89f9-9259b9c8bc77","Type":"ContainerStarted","Data":"833e970fcbe394a8b7afda4b830ef91e1a267c55c46a2500ef022752c896339d"}
Sep 12 00:59:28.653248 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:28.652855    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq" event={"ID":"682f03ec-d30f-4129-89f9-9259b9c8bc77","Type":"ContainerStarted","Data":"5d67d6423616860c23703880f7171dc5126885a54b7ee064a1115c90a7d1fe8d"}
Sep 12 00:59:28.670442 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:28.670385    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq" podStartSLOduration=1.67036871 podStartE2EDuration="1.67036871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:59:28.66998574 +0000 UTC m=+973.230437405" watchObservedRunningTime="2026-09-12 00:59:28.67036871 +0000 UTC m=+973.230820377"
Sep 12 00:59:37.439894 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.439822    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"]
Sep 12 00:59:37.443783 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.443758    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.446385 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.446361    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f19fd1\"/\"premium-w4-f19fd1-kserve-self-signed-certs\""
Sep 12 00:59:37.454676 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.454652    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"]
Sep 12 00:59:37.463128 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.463099    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:37.576563 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.576521    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql5s4\" (UniqueName: \"kubernetes.io/projected/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-kube-api-access-ql5s4\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.576728 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.576585    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-tmp-dir\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.576728 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.576653    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-home\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.576804 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.576716    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-tls-certs\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.576804 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.576757    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-model-cache\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.576804 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.576791    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-dshm\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.656542 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.656497    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:37.656680 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.656558    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:37.669563 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.669535    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:37.677741 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.677718    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-dshm\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.677815 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.677778    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5s4\" (UniqueName: \"kubernetes.io/projected/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-kube-api-access-ql5s4\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.677854 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.677825    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-tmp-dir\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.677854 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.677848    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-home\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.677930 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.677885    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-tls-certs\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.677930 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.677907    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-model-cache\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.678193 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.678172    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-tmp-dir\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.678265 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.678198    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-model-cache\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.678319 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.678269    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-home\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.680198 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.680177    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-dshm\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.680305 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.680287    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-tls-certs\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.685631 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.685590    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql5s4\" (UniqueName: \"kubernetes.io/projected/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-kube-api-access-ql5s4\") pod \"premium-w4-f19fd1-kserve-589c4df7bc-nsr78\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") " pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:37.709662 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.709607    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 00:59:37.754475 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:37.754448    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:38.039477 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:38.039450    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"]
Sep 12 00:59:38.040917 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:59:38.040888    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda20ac225_20b4_4ec7_8a8b_c2c47a3bb747.slice/crio-d93967aaacd4f4a6301ec40077783c4bfdb3f0c804b8fe978d07a0f8f739661d WatchSource:0}: Error finding container d93967aaacd4f4a6301ec40077783c4bfdb3f0c804b8fe978d07a0f8f739661d: Status 404 returned error can't find the container with id d93967aaacd4f4a6301ec40077783c4bfdb3f0c804b8fe978d07a0f8f739661d
Sep 12 00:59:38.705108 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:38.705062    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78" event={"ID":"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747","Type":"ContainerStarted","Data":"444764f0fc6f03fe6b4cf72a7027724cfe5f73645c5b9410a8e38a25ff9725a9"}
Sep 12 00:59:38.705469 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:38.705114    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78" event={"ID":"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747","Type":"ContainerStarted","Data":"d93967aaacd4f4a6301ec40077783c4bfdb3f0c804b8fe978d07a0f8f739661d"}
Sep 12 00:59:38.725322 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:38.725270    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78" podStartSLOduration=1.7252562569999998 podStartE2EDuration="1.725256257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:59:38.722679938 +0000 UTC m=+983.283131621" watchObservedRunningTime="2026-09-12 00:59:38.725256257 +0000 UTC m=+983.285707924"
Sep 12 00:59:39.635330 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:39.635297    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:39.914244 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:39.914208    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:40.420391 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:40.420362    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:40.683580 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:40.683503    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:40.785689 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:40.785663    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:41.059571 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:41.059502    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:43.384657 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:43.384624    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:43.941203 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:43.941170    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:44.059175 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:44.059140    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:44.390145 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:44.390074    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:45.293491 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:45.293460    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:46.148968 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:46.148922    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:46.473248 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:46.473216    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:47.064567 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:47.064536    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:47.754631 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:47.754603    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:47.755074 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:47.754653    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:47.767346 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:47.767322    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:47.883370 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:47.883338    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"]
Sep 12 00:59:47.886995 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:47.886976    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:47.889539 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:47.889520    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-8a4fbb-serving-cert\""
Sep 12 00:59:47.897109 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:47.896906    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"]
Sep 12 00:59:47.941220 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:47.941189    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:47.976254 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:47.976224    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-maas-api-tls\") pod \"maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:47.976463 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:47.976442    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-openshift-service-ca\") pod \"maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:47.976613 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:47.976598    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-metrics-certs\") pod \"maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:47.976794 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:47.976779    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwlvf\" (UniqueName: \"kubernetes.io/projected/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-kube-api-access-qwlvf\") pod \"maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:48.077902 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.077824    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwlvf\" (UniqueName: \"kubernetes.io/projected/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-kube-api-access-qwlvf\") pod \"maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:48.078060 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.077917    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-maas-api-tls\") pod \"maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:48.078060 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.077974    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-openshift-service-ca\") pod \"maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:48.078060 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.078017    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-metrics-certs\") pod \"maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:48.078634 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.078612    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-openshift-service-ca\") pod \"maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:48.080556 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.080528    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:48.081009 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.080986    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-maas-api-tls\") pod \"maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:48.081088 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.080991    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-metrics-certs\") pod \"maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:48.087742 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.087720    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwlvf\" (UniqueName: \"kubernetes.io/projected/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-kube-api-access-qwlvf\") pod \"maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:48.173351 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.173316    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:48.201435 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.201401    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:48.282433 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.282404    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:48.374200 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.374131    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:48.475993 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.475934    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"]
Sep 12 00:59:48.478091 ip-10-0-139-121 kubenswrapper[2567]: W0912 00:59:48.478064    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502ebd8d_e374_4f1c_9ae0_27d25af6aa7a.slice/crio-b1042ec72387ee27ba39e043e318147425696e7ad69d5ec43134c3b62d4b9d13 WatchSource:0}: Error finding container b1042ec72387ee27ba39e043e318147425696e7ad69d5ec43134c3b62d4b9d13: Status 404 returned error can't find the container with id b1042ec72387ee27ba39e043e318147425696e7ad69d5ec43134c3b62d4b9d13
Sep 12 00:59:48.479524 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.479506    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:59:48.585285 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.585247    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:48.679849 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.679814    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:48.755332 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.755304    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f" event={"ID":"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a","Type":"ContainerStarted","Data":"b1042ec72387ee27ba39e043e318147425696e7ad69d5ec43134c3b62d4b9d13"}
Sep 12 00:59:48.766561 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:48.766537    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 00:59:49.761529 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:49.761497    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f" event={"ID":"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a","Type":"ContainerStarted","Data":"a911985e801582ec3b9d1e85d9b6faeabcf0ce73e97d9e5759c10adeff7b79b6"}
Sep 12 00:59:49.762025 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:49.761713    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:49.782604 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:49.782561    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f" podStartSLOduration=2.340494048 podStartE2EDuration="2.782545203s" totalImagesPullingTime="442.051155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:59:49.781214081 +0000 UTC m=+994.341665775" watchObservedRunningTime="2026-09-12 00:59:49.782545203 +0000 UTC m=+994.342996871"
Sep 12 00:59:55.772160 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:55.772129    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 00:59:58.877719 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:58.877668    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:59.683267 ip-10-0-139-121 kubenswrapper[2567]: I0912 00:59:59.683232    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:05.497085 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:05.497049    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:05.622193 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:05.622164    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:05.768537 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:05.768441    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:05.870259 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:05.870202    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:06.481960 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:06.481906    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:06.566049 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:06.565998    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:06.671938 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:06.671898    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:08.132206 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:08.132166    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:10.397971 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:10.397914    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:11.316769 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:11.316735    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:12.255299 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:12.255263    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:13.214508 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:13.214464    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:13.993653 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:13.993614    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:14.102250 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:14.102204    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:14.254731 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:14.254649    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:14.354766 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:14.354714    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:14.901533 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:14.901483    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:15.002344 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:15.002314    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:21.056060 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:21.056022    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:21.157812 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:21.157781    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:21.311708 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:21.311628    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:21.628094 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:21.627989    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:21.823145 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:21.823117    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:27.954696 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:27.954661    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:28.037149 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:28.037124    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:28.138084 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:28.138042    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:28.931546 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:28.931514    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:29.808110 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:29.808075    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:30.598627 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:30.598581    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:31.370127 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:31.370099    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:32.259009 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.258975    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"]
Sep 12 01:00:32.264183 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.264160    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:32.266843 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.266825    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9eca3408-serving-cert\""
Sep 12 01:00:32.274179 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.274156    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"]
Sep 12 01:00:32.290015 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.289986    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:32.388266 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.388229    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:32.408445 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.408411    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd80673b-db3d-4983-8e42-5195c724e3ca-maas-api-tls\") pod \"maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:32.408445 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.408444    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd80673b-db3d-4983-8e42-5195c724e3ca-metrics-certs\") pod \"maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:32.408624 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.408471    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd80673b-db3d-4983-8e42-5195c724e3ca-openshift-service-ca\") pod \"maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:32.408624 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.408615    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj7dx\" (UniqueName: \"kubernetes.io/projected/bd80673b-db3d-4983-8e42-5195c724e3ca-kube-api-access-vj7dx\") pod \"maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:32.509172 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.509088    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj7dx\" (UniqueName: \"kubernetes.io/projected/bd80673b-db3d-4983-8e42-5195c724e3ca-kube-api-access-vj7dx\") pod \"maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:32.509172 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.509148    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd80673b-db3d-4983-8e42-5195c724e3ca-maas-api-tls\") pod \"maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:32.509172 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.509164    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd80673b-db3d-4983-8e42-5195c724e3ca-metrics-certs\") pod \"maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:32.509398 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.509190    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd80673b-db3d-4983-8e42-5195c724e3ca-openshift-service-ca\") pod \"maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:32.510067 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.509983    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd80673b-db3d-4983-8e42-5195c724e3ca-openshift-service-ca\") pod \"maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:32.510430 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.510405    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:32.511779 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.511756    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd80673b-db3d-4983-8e42-5195c724e3ca-metrics-certs\") pod \"maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:32.511859 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.511790    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd80673b-db3d-4983-8e42-5195c724e3ca-maas-api-tls\") pod \"maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:32.517750 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.517729    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj7dx\" (UniqueName: \"kubernetes.io/projected/bd80673b-db3d-4983-8e42-5195c724e3ca-kube-api-access-vj7dx\") pod \"maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:32.577052 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.577025    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:32.852413 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.852385    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"]
Sep 12 01:00:32.854130 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:00:32.854088    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd80673b_db3d_4983_8e42_5195c724e3ca.slice/crio-e386a911611269bb7d1bebb7577d947cce65693576c450d07cbd6333859126ad WatchSource:0}: Error finding container e386a911611269bb7d1bebb7577d947cce65693576c450d07cbd6333859126ad: Status 404 returned error can't find the container with id e386a911611269bb7d1bebb7577d947cce65693576c450d07cbd6333859126ad
Sep 12 01:00:32.879861 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.879823    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:32.978234 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.978193    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:32.980586 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:32.980560    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh" event={"ID":"bd80673b-db3d-4983-8e42-5195c724e3ca","Type":"ContainerStarted","Data":"e386a911611269bb7d1bebb7577d947cce65693576c450d07cbd6333859126ad"}
Sep 12 01:00:33.986839 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:33.986808    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh" event={"ID":"bd80673b-db3d-4983-8e42-5195c724e3ca","Type":"ContainerStarted","Data":"a15a507b9ca5614f7fcf1d748d7968370285e9c094bc4e3f406634fa409f97e8"}
Sep 12 01:00:33.987289 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:33.986910    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:34.003606 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:34.003555    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh" podStartSLOduration=1.578427853 podStartE2EDuration="2.003537487s" totalImagesPullingTime="425.109634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:00:34.002928967 +0000 UTC m=+1038.563380651" watchObservedRunningTime="2026-09-12 01:00:34.003537487 +0000 UTC m=+1038.563989157"
Sep 12 01:00:39.998597 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:39.998550    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:00:54.489396 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:54.489364    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:55.259376 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:55.259335    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:56.115644 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:56.115613    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:56.652350 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:56.652309    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:56.845531 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:56.845443    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:57.527871 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:57.527818    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:57.929014 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:57.928983    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:58.495424 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:58.495383    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:58.598628 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:58.598592    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:58.697767 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:58.697734    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:59.066371 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:59.066338    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:00:59.433356 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:00:59.433323    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:04.725429 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:04.725397    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:04.825196 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:04.825165    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:06.371668 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:06.371639    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:06.687429 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:06.687350    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:07.395184 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:07.395153    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:08.195551 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:08.195517    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:08.915199 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:08.915164    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:09.624941 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:09.624901    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:10.503205 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:10.503163    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:15.599075 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:15.599042    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"]
Sep 12 01:01:15.599474 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:15.599309    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh" podUID="bd80673b-db3d-4983-8e42-5195c724e3ca" containerName="maas-api" containerID="cri-o://a15a507b9ca5614f7fcf1d748d7968370285e9c094bc4e3f406634fa409f97e8" gracePeriod=30
Sep 12 01:01:15.638158 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:15.638124    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:15.865016 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:15.864934    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:01:15.914481 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:15.914452    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:15.935677 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:15.935651    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vj7dx\" (UniqueName: \"kubernetes.io/projected/bd80673b-db3d-4983-8e42-5195c724e3ca-kube-api-access-vj7dx\") pod \"bd80673b-db3d-4983-8e42-5195c724e3ca\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") "
Sep 12 01:01:15.935782 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:15.935687    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd80673b-db3d-4983-8e42-5195c724e3ca-metrics-certs\") pod \"bd80673b-db3d-4983-8e42-5195c724e3ca\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") "
Sep 12 01:01:15.935782 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:15.935716    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd80673b-db3d-4983-8e42-5195c724e3ca-maas-api-tls\") pod \"bd80673b-db3d-4983-8e42-5195c724e3ca\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") "
Sep 12 01:01:15.935875 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:15.935830    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd80673b-db3d-4983-8e42-5195c724e3ca-openshift-service-ca\") pod \"bd80673b-db3d-4983-8e42-5195c724e3ca\" (UID: \"bd80673b-db3d-4983-8e42-5195c724e3ca\") "
Sep 12 01:01:15.936254 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:15.936223    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd80673b-db3d-4983-8e42-5195c724e3ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bd80673b-db3d-4983-8e42-5195c724e3ca" (UID: "bd80673b-db3d-4983-8e42-5195c724e3ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:01:15.937970 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:15.937930    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd80673b-db3d-4983-8e42-5195c724e3ca-kube-api-access-vj7dx" (OuterVolumeSpecName: "kube-api-access-vj7dx") pod "bd80673b-db3d-4983-8e42-5195c724e3ca" (UID: "bd80673b-db3d-4983-8e42-5195c724e3ca"). InnerVolumeSpecName "kube-api-access-vj7dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:01:15.938075 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:15.937972    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd80673b-db3d-4983-8e42-5195c724e3ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bd80673b-db3d-4983-8e42-5195c724e3ca" (UID: "bd80673b-db3d-4983-8e42-5195c724e3ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:01:15.938119 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:15.938073    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd80673b-db3d-4983-8e42-5195c724e3ca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bd80673b-db3d-4983-8e42-5195c724e3ca" (UID: "bd80673b-db3d-4983-8e42-5195c724e3ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:01:15.991930 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:15.991906    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:16.037685 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.037658    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj7dx\" (UniqueName: \"kubernetes.io/projected/bd80673b-db3d-4983-8e42-5195c724e3ca-kube-api-access-vj7dx\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:16.037685 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.037682    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd80673b-db3d-4983-8e42-5195c724e3ca-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:16.037846 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.037704    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd80673b-db3d-4983-8e42-5195c724e3ca-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:16.037846 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.037717    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd80673b-db3d-4983-8e42-5195c724e3ca-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:16.197156 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.197124    2567 generic.go:356] "Generic (PLEG): container finished" podID="bd80673b-db3d-4983-8e42-5195c724e3ca" containerID="a15a507b9ca5614f7fcf1d748d7968370285e9c094bc4e3f406634fa409f97e8" exitCode=0
Sep 12 01:01:16.197357 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.197173    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"
Sep 12 01:01:16.197357 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.197200    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh" event={"ID":"bd80673b-db3d-4983-8e42-5195c724e3ca","Type":"ContainerDied","Data":"a15a507b9ca5614f7fcf1d748d7968370285e9c094bc4e3f406634fa409f97e8"}
Sep 12 01:01:16.197357 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.197262    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh" event={"ID":"bd80673b-db3d-4983-8e42-5195c724e3ca","Type":"ContainerDied","Data":"e386a911611269bb7d1bebb7577d947cce65693576c450d07cbd6333859126ad"}
Sep 12 01:01:16.197357 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.197292    2567 scope.go:117] "RemoveContainer" containerID="a15a507b9ca5614f7fcf1d748d7968370285e9c094bc4e3f406634fa409f97e8"
Sep 12 01:01:16.204492 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.204441    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:16.207419 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.207403    2567 scope.go:117] "RemoveContainer" containerID="a15a507b9ca5614f7fcf1d748d7968370285e9c094bc4e3f406634fa409f97e8"
Sep 12 01:01:16.207666 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:01:16.207642    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a15a507b9ca5614f7fcf1d748d7968370285e9c094bc4e3f406634fa409f97e8\": container with ID starting with a15a507b9ca5614f7fcf1d748d7968370285e9c094bc4e3f406634fa409f97e8 not found: ID does not exist" containerID="a15a507b9ca5614f7fcf1d748d7968370285e9c094bc4e3f406634fa409f97e8"
Sep 12 01:01:16.207760 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.207674    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15a507b9ca5614f7fcf1d748d7968370285e9c094bc4e3f406634fa409f97e8"} err="failed to get container status \"a15a507b9ca5614f7fcf1d748d7968370285e9c094bc4e3f406634fa409f97e8\": rpc error: code = NotFound desc = could not find container \"a15a507b9ca5614f7fcf1d748d7968370285e9c094bc4e3f406634fa409f97e8\": container with ID starting with a15a507b9ca5614f7fcf1d748d7968370285e9c094bc4e3f406634fa409f97e8 not found: ID does not exist"
Sep 12 01:01:16.214606 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.214585    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"]
Sep 12 01:01:16.218760 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.218741    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9eca3408-5b4df4fb-trsmh"]
Sep 12 01:01:16.305822 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:16.305800    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:17.101487 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:17.101456    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:17.888729 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:17.888697    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:17.974093 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:17.974067    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd80673b-db3d-4983-8e42-5195c724e3ca" path="/var/lib/kubelet/pods/bd80673b-db3d-4983-8e42-5195c724e3ca/volumes"
Sep 12 01:01:18.608923 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:18.608891    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:19.383932 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:19.383898    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:20.050491 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.050453    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"]
Sep 12 01:01:20.050859 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.050837    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd80673b-db3d-4983-8e42-5195c724e3ca" containerName="maas-api"
Sep 12 01:01:20.050859 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.050848    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd80673b-db3d-4983-8e42-5195c724e3ca" containerName="maas-api"
Sep 12 01:01:20.050935 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.050922    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd80673b-db3d-4983-8e42-5195c724e3ca" containerName="maas-api"
Sep 12 01:01:20.055568 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.055549    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.058884 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.058862    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f19fd1\"/\"e2e-conflict-a-f6b886e1-kserve-self-signed-certs\""
Sep 12 01:01:20.072334 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.072311    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"]
Sep 12 01:01:20.116708 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.116684    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:20.175355 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.175326    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-home\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.175355 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.175358    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-dshm\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.175522 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.175377    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s625s\" (UniqueName: \"kubernetes.io/projected/b5245f7f-74f3-4ec5-bff4-663e759e7385-kube-api-access-s625s\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.175522 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.175471    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-tmp-dir\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.175522 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.175506    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-model-cache\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.175628 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.175535    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5245f7f-74f3-4ec5-bff4-663e759e7385-tls-certs\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.218088 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.218056    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:20.276488 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.276445    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-tmp-dir\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.276632 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.276500    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-model-cache\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.276632 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.276525    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5245f7f-74f3-4ec5-bff4-663e759e7385-tls-certs\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.276632 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.276582    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-home\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.276632 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.276597    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-dshm\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.276632 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.276613    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s625s\" (UniqueName: \"kubernetes.io/projected/b5245f7f-74f3-4ec5-bff4-663e759e7385-kube-api-access-s625s\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.276941 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.276858    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-tmp-dir\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.276941 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.276912    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-model-cache\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.277093 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.277025    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-home\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.278963 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.278925    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-dshm\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.279239 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.279221    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5245f7f-74f3-4ec5-bff4-663e759e7385-tls-certs\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.285975 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.285942    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s625s\" (UniqueName: \"kubernetes.io/projected/b5245f7f-74f3-4ec5-bff4-663e759e7385-kube-api-access-s625s\") pod \"e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") " pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.365227 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.365159    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:20.661869 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:20.661844    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"]
Sep 12 01:01:20.664469 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:01:20.664443    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5245f7f_74f3_4ec5_bff4_663e759e7385.slice/crio-8054ee297b7cb0484e305baf09ec9a0149d9d15e841fcc57dc37556db353802b WatchSource:0}: Error finding container 8054ee297b7cb0484e305baf09ec9a0149d9d15e841fcc57dc37556db353802b: Status 404 returned error can't find the container with id 8054ee297b7cb0484e305baf09ec9a0149d9d15e841fcc57dc37556db353802b
Sep 12 01:01:21.224680 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:21.224642    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f" event={"ID":"b5245f7f-74f3-4ec5-bff4-663e759e7385","Type":"ContainerStarted","Data":"1d94e69896c4f44470c7ccb19d8917df2f9950e32bf1ef98234b1b7ea394f69f"}
Sep 12 01:01:21.225114 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:21.224689    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f" event={"ID":"b5245f7f-74f3-4ec5-bff4-663e759e7385","Type":"ContainerStarted","Data":"8054ee297b7cb0484e305baf09ec9a0149d9d15e841fcc57dc37556db353802b"}
Sep 12 01:01:21.243489 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:21.243440    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f" podStartSLOduration=1.24342644 podStartE2EDuration="1.24342644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:01:21.242419661 +0000 UTC m=+1085.802871336" watchObservedRunningTime="2026-09-12 01:01:21.24342644 +0000 UTC m=+1085.803878109"
Sep 12 01:01:22.017647 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:22.017613    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:22.079764 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:22.079737    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:22.377174 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:22.377099    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:22.645375 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:22.645307    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:22.744018 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:22.743990    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:22.870065 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:22.870033    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:28.631494 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:28.631463    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:28.737000 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:28.736968    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:28.808109 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:28.808075    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:28.925534 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:28.925505    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:29.025380 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:29.025354    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:29.128395 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:29.128364    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:30.365509 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:30.365462    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:30.365509 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:30.365525    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:30.378076 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:30.378056    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:31.257675 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:31.257642    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:31.284994 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:31.284962    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:01:31.345864 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:31.345831    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:32.143882 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:32.143848    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:32.297507 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:32.297475    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:32.997462 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:32.997431    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:33.686445 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:33.686407    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:34.614232 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:34.614202    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:35.261969 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:35.261909    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:35.365233 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:35.365203    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:36.077678 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:36.077649    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:36.178435 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:36.178403    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:36.278286 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:36.278258    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:36.498221 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:36.498185    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:36.788685 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:36.788604    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:37.164453 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:37.164422    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:37.512559 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:37.512473    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:38.295570 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:38.295534    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:39.097132 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:39.097096    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:39.472838 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:39.472799    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:39.561480 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:39.561450    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:40.125990 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:40.125934    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:41.117011 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:41.116982    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:41.926846 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:41.926807    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:42.511490 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.511452    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"]
Sep 12 01:01:42.511716 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.511695    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f" podUID="502ebd8d-e374-4f1c-9ae0-27d25af6aa7a" containerName="maas-api" containerID="cri-o://a911985e801582ec3b9d1e85d9b6faeabcf0ce73e97d9e5759c10adeff7b79b6" gracePeriod=30
Sep 12 01:01:42.526983 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.526935    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:42.612394 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.612359    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:42.681459 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.681421    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:42.760458 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.760429    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:42.763453 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.763404    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 01:01:42.788370 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.788349    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-maas-api-tls\") pod \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") "
Sep 12 01:01:42.788519 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.788390    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-metrics-certs\") pod \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") "
Sep 12 01:01:42.788519 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.788444    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-openshift-service-ca\") pod \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") "
Sep 12 01:01:42.788519 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.788469    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwlvf\" (UniqueName: \"kubernetes.io/projected/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-kube-api-access-qwlvf\") pod \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\" (UID: \"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a\") "
Sep 12 01:01:42.789247 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.789221    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "502ebd8d-e374-4f1c-9ae0-27d25af6aa7a" (UID: "502ebd8d-e374-4f1c-9ae0-27d25af6aa7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:01:42.791037 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.791013    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-kube-api-access-qwlvf" (OuterVolumeSpecName: "kube-api-access-qwlvf") pod "502ebd8d-e374-4f1c-9ae0-27d25af6aa7a" (UID: "502ebd8d-e374-4f1c-9ae0-27d25af6aa7a"). InnerVolumeSpecName "kube-api-access-qwlvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:01:42.791456 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.791435    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "502ebd8d-e374-4f1c-9ae0-27d25af6aa7a" (UID: "502ebd8d-e374-4f1c-9ae0-27d25af6aa7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:01:42.791600 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.791575    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "502ebd8d-e374-4f1c-9ae0-27d25af6aa7a" (UID: "502ebd8d-e374-4f1c-9ae0-27d25af6aa7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:01:42.825038 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.825010    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:42.890150 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.890126    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:42.890150 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.890150    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:42.890316 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.890160    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:42.890316 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:42.890169    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwlvf\" (UniqueName: \"kubernetes.io/projected/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a-kube-api-access-qwlvf\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:43.087619 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.087540    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:43.187263 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.187228    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:43.337349 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.337319    2567 generic.go:356] "Generic (PLEG): container finished" podID="502ebd8d-e374-4f1c-9ae0-27d25af6aa7a" containerID="a911985e801582ec3b9d1e85d9b6faeabcf0ce73e97d9e5759c10adeff7b79b6" exitCode=0
Sep 12 01:01:43.337506 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.337360    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"
Sep 12 01:01:43.337506 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.337403    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f" event={"ID":"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a","Type":"ContainerDied","Data":"a911985e801582ec3b9d1e85d9b6faeabcf0ce73e97d9e5759c10adeff7b79b6"}
Sep 12 01:01:43.337506 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.337454    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f" event={"ID":"502ebd8d-e374-4f1c-9ae0-27d25af6aa7a","Type":"ContainerDied","Data":"b1042ec72387ee27ba39e043e318147425696e7ad69d5ec43134c3b62d4b9d13"}
Sep 12 01:01:43.337506 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.337478    2567 scope.go:117] "RemoveContainer" containerID="a911985e801582ec3b9d1e85d9b6faeabcf0ce73e97d9e5759c10adeff7b79b6"
Sep 12 01:01:43.346815 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.346801    2567 scope.go:117] "RemoveContainer" containerID="a911985e801582ec3b9d1e85d9b6faeabcf0ce73e97d9e5759c10adeff7b79b6"
Sep 12 01:01:43.347088 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:01:43.347069    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a911985e801582ec3b9d1e85d9b6faeabcf0ce73e97d9e5759c10adeff7b79b6\": container with ID starting with a911985e801582ec3b9d1e85d9b6faeabcf0ce73e97d9e5759c10adeff7b79b6 not found: ID does not exist" containerID="a911985e801582ec3b9d1e85d9b6faeabcf0ce73e97d9e5759c10adeff7b79b6"
Sep 12 01:01:43.347138 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.347099    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a911985e801582ec3b9d1e85d9b6faeabcf0ce73e97d9e5759c10adeff7b79b6"} err="failed to get container status \"a911985e801582ec3b9d1e85d9b6faeabcf0ce73e97d9e5759c10adeff7b79b6\": rpc error: code = NotFound desc = could not find container \"a911985e801582ec3b9d1e85d9b6faeabcf0ce73e97d9e5759c10adeff7b79b6\": container with ID starting with a911985e801582ec3b9d1e85d9b6faeabcf0ce73e97d9e5759c10adeff7b79b6 not found: ID does not exist"
Sep 12 01:01:43.358860 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.358838    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"]
Sep 12 01:01:43.362204 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.362183    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8a4fbb-847448dfd5-n6r5f"]
Sep 12 01:01:43.875302 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.875272    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"]
Sep 12 01:01:43.875691 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.875677    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="502ebd8d-e374-4f1c-9ae0-27d25af6aa7a" containerName="maas-api"
Sep 12 01:01:43.875759 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.875693    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="502ebd8d-e374-4f1c-9ae0-27d25af6aa7a" containerName="maas-api"
Sep 12 01:01:43.875815 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.875767    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="502ebd8d-e374-4f1c-9ae0-27d25af6aa7a" containerName="maas-api"
Sep 12 01:01:43.880375 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.880352    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:43.882858 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.882838    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-8e62c2-serving-cert\""
Sep 12 01:01:43.894363 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.894336    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"]
Sep 12 01:01:43.917386 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.917363    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:43.973651 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:43.973626    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="502ebd8d-e374-4f1c-9ae0-27d25af6aa7a" path="/var/lib/kubelet/pods/502ebd8d-e374-4f1c-9ae0-27d25af6aa7a/volumes"
Sep 12 01:01:44.001916 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.001892    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c7e3cbe-53ef-4a7c-b217-de40db448969-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:44.002090 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.001968    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7e3cbe-53ef-4a7c-b217-de40db448969-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:44.002090 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.001997    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9z9p\" (UniqueName: \"kubernetes.io/projected/4c7e3cbe-53ef-4a7c-b217-de40db448969-kube-api-access-d9z9p\") pod \"maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:44.002090 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.002019    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c7e3cbe-53ef-4a7c-b217-de40db448969-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:44.015177 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.015151    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:44.076084 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.076050    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:44.103524 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.103497    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c7e3cbe-53ef-4a7c-b217-de40db448969-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:44.103872 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.103547    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7e3cbe-53ef-4a7c-b217-de40db448969-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:44.103872 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.103719    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9z9p\" (UniqueName: \"kubernetes.io/projected/4c7e3cbe-53ef-4a7c-b217-de40db448969-kube-api-access-d9z9p\") pod \"maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:44.103872 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.103743    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c7e3cbe-53ef-4a7c-b217-de40db448969-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:44.104426 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.104407    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c7e3cbe-53ef-4a7c-b217-de40db448969-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:44.106135 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.106115    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c7e3cbe-53ef-4a7c-b217-de40db448969-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:44.106225 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.106158    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7e3cbe-53ef-4a7c-b217-de40db448969-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:44.111410 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.111388    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9z9p\" (UniqueName: \"kubernetes.io/projected/4c7e3cbe-53ef-4a7c-b217-de40db448969-kube-api-access-d9z9p\") pod \"maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:44.191705 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.191680    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:44.474011 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.473982    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"]
Sep 12 01:01:44.476251 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:01:44.476220    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7e3cbe_53ef_4a7c_b217_de40db448969.slice/crio-82f6f375c36d2a8a9aea6de85d92402843e81aa0b058f14334f7c55695895cb2 WatchSource:0}: Error finding container 82f6f375c36d2a8a9aea6de85d92402843e81aa0b058f14334f7c55695895cb2: Status 404 returned error can't find the container with id 82f6f375c36d2a8a9aea6de85d92402843e81aa0b058f14334f7c55695895cb2
Sep 12 01:01:44.498124 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.498096    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:44.607898 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.607864    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:44.918107 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:44.918075    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:45.350423 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:45.350348    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5" event={"ID":"4c7e3cbe-53ef-4a7c-b217-de40db448969","Type":"ContainerStarted","Data":"b7750204fc387b7e79602576bb54e4cd4ba02b3036849ad0024894f0bffed52c"}
Sep 12 01:01:45.350423 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:45.350399    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5" event={"ID":"4c7e3cbe-53ef-4a7c-b217-de40db448969","Type":"ContainerStarted","Data":"82f6f375c36d2a8a9aea6de85d92402843e81aa0b058f14334f7c55695895cb2"}
Sep 12 01:01:45.350803 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:45.350478    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:45.369316 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:45.369274    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5" podStartSLOduration=1.841285458 podStartE2EDuration="2.369254834s" totalImagesPullingTime="527.969376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:01:45.367136977 +0000 UTC m=+1109.927588660" watchObservedRunningTime="2026-09-12 01:01:45.369254834 +0000 UTC m=+1109.929706562"
Sep 12 01:01:45.786556 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:45.786523    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:46.624670 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:46.624636    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:47.416473 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:47.416441    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:48.288896 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:48.288859    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:49.012474 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:49.012447    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:49.266806 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:49.266716    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:49.366365 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:49.366332    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:49.529109 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:49.529025    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:49.968408 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:49.968374    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:50.266880 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:50.266796    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:50.592582 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:50.592506    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:51.361207 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:51.361177    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:01:51.842559 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:51.842529    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:51.947524 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:51.947493    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:52.043526 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:52.043495    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:52.483964 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:52.483915    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:01:56.280942 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:01:56.280909    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:03.201130 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:03.201091    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:04.076047 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:04.076017    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:04.589995 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:04.589968    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:05.505563 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:05.505533    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:06.181680 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:06.181652    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:07.064562 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:07.064527    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:07.749095 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:07.749066    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:08.509938 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:08.509904    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:09.381797 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:09.381767    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:09.482907 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:09.482866    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:09.776108 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:09.776061    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:10.093553 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:10.093480    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:10.208069 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:10.208029    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:10.658697 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:10.658663    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:10.926751 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:10.926668    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:11.895807 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:11.895775    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:12.687732 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:12.687699    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:12.822994 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:12.822963    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:12.919886 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:12.919857    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:13.698933 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:13.698898    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:14.414100 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:14.414068    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:15.017826 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.017795    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"]
Sep 12 01:02:15.018094 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.018033    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt" podUID="21bbe704-fc50-475e-ada4-5bf62d5bb62d" containerName="maas-api" containerID="cri-o://4eb0f4d4e65f1c1754ad25b37c3329c6e3d9a2e2e9c87f94cda5d3522f31117f" gracePeriod=30
Sep 12 01:02:15.052484 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.052455    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:15.136085 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.136026    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:15.213935 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.213903    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:15.278127 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.278067    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 01:02:15.281463 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.281439    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:15.321486 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.321459    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-maas-api-tls\") pod \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") "
Sep 12 01:02:15.321644 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.321566    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24xht\" (UniqueName: \"kubernetes.io/projected/21bbe704-fc50-475e-ada4-5bf62d5bb62d-kube-api-access-24xht\") pod \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") "
Sep 12 01:02:15.321644 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.321592    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bbe704-fc50-475e-ada4-5bf62d5bb62d-openshift-service-ca\") pod \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") "
Sep 12 01:02:15.321644 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.321617    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-metrics-certs\") pod \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\" (UID: \"21bbe704-fc50-475e-ada4-5bf62d5bb62d\") "
Sep 12 01:02:15.322030 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.322006    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21bbe704-fc50-475e-ada4-5bf62d5bb62d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "21bbe704-fc50-475e-ada4-5bf62d5bb62d" (UID: "21bbe704-fc50-475e-ada4-5bf62d5bb62d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:02:15.323781 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.323753    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "21bbe704-fc50-475e-ada4-5bf62d5bb62d" (UID: "21bbe704-fc50-475e-ada4-5bf62d5bb62d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:02:15.323877 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.323812    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bbe704-fc50-475e-ada4-5bf62d5bb62d-kube-api-access-24xht" (OuterVolumeSpecName: "kube-api-access-24xht") pod "21bbe704-fc50-475e-ada4-5bf62d5bb62d" (UID: "21bbe704-fc50-475e-ada4-5bf62d5bb62d"). InnerVolumeSpecName "kube-api-access-24xht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:02:15.323877 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.323850    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "21bbe704-fc50-475e-ada4-5bf62d5bb62d" (UID: "21bbe704-fc50-475e-ada4-5bf62d5bb62d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:02:15.352010 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.351976    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:15.422652 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.422603    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24xht\" (UniqueName: \"kubernetes.io/projected/21bbe704-fc50-475e-ada4-5bf62d5bb62d-kube-api-access-24xht\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:15.422652 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.422628    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bbe704-fc50-475e-ada4-5bf62d5bb62d-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:15.422652 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.422643    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:15.422652 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.422655    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21bbe704-fc50-475e-ada4-5bf62d5bb62d-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:15.500881 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.500849    2567 generic.go:356] "Generic (PLEG): container finished" podID="21bbe704-fc50-475e-ada4-5bf62d5bb62d" containerID="4eb0f4d4e65f1c1754ad25b37c3329c6e3d9a2e2e9c87f94cda5d3522f31117f" exitCode=0
Sep 12 01:02:15.501071 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.500893    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"
Sep 12 01:02:15.501071 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.500890    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt" event={"ID":"21bbe704-fc50-475e-ada4-5bf62d5bb62d","Type":"ContainerDied","Data":"4eb0f4d4e65f1c1754ad25b37c3329c6e3d9a2e2e9c87f94cda5d3522f31117f"}
Sep 12 01:02:15.501071 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.500968    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt" event={"ID":"21bbe704-fc50-475e-ada4-5bf62d5bb62d","Type":"ContainerDied","Data":"7f777a49a6164291ad212df5d3f4e288dacdc21062557c3b1193f09e88a98570"}
Sep 12 01:02:15.501071 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.501001    2567 scope.go:117] "RemoveContainer" containerID="4eb0f4d4e65f1c1754ad25b37c3329c6e3d9a2e2e9c87f94cda5d3522f31117f"
Sep 12 01:02:15.510297 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.510282    2567 scope.go:117] "RemoveContainer" containerID="4eb0f4d4e65f1c1754ad25b37c3329c6e3d9a2e2e9c87f94cda5d3522f31117f"
Sep 12 01:02:15.510562 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:02:15.510546    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb0f4d4e65f1c1754ad25b37c3329c6e3d9a2e2e9c87f94cda5d3522f31117f\": container with ID starting with 4eb0f4d4e65f1c1754ad25b37c3329c6e3d9a2e2e9c87f94cda5d3522f31117f not found: ID does not exist" containerID="4eb0f4d4e65f1c1754ad25b37c3329c6e3d9a2e2e9c87f94cda5d3522f31117f"
Sep 12 01:02:15.510601 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.510573    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb0f4d4e65f1c1754ad25b37c3329c6e3d9a2e2e9c87f94cda5d3522f31117f"} err="failed to get container status \"4eb0f4d4e65f1c1754ad25b37c3329c6e3d9a2e2e9c87f94cda5d3522f31117f\": rpc error: code = NotFound desc = could not find container \"4eb0f4d4e65f1c1754ad25b37c3329c6e3d9a2e2e9c87f94cda5d3522f31117f\": container with ID starting with 4eb0f4d4e65f1c1754ad25b37c3329c6e3d9a2e2e9c87f94cda5d3522f31117f not found: ID does not exist"
Sep 12 01:02:15.522688 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.522665    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"]
Sep 12 01:02:15.526330 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.526306    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cee89-5c75db58f8-ngntt"]
Sep 12 01:02:15.612841 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.612777    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:15.973780 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:15.973751    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21bbe704-fc50-475e-ada4-5bf62d5bb62d" path="/var/lib/kubelet/pods/21bbe704-fc50-475e-ada4-5bf62d5bb62d/volumes"
Sep 12 01:02:16.213455 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:16.213415    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:16.889468 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:16.889426    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"]
Sep 12 01:02:16.890055 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:16.890036    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21bbe704-fc50-475e-ada4-5bf62d5bb62d" containerName="maas-api"
Sep 12 01:02:16.890055 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:16.890056    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="21bbe704-fc50-475e-ada4-5bf62d5bb62d" containerName="maas-api"
Sep 12 01:02:16.890219 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:16.890158    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="21bbe704-fc50-475e-ada4-5bf62d5bb62d" containerName="maas-api"
Sep 12 01:02:16.895290 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:16.895266    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:16.897988 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:16.897967    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-18b9d7-serving-cert\""
Sep 12 01:02:16.909226 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:16.909200    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"]
Sep 12 01:02:16.926323 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:16.926292    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:16.938086 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:16.938060    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:16.938231 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:16.938173    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:16.938298 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:16.938246    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9d8\" (UniqueName: \"kubernetes.io/projected/e6912507-9a44-4218-97ff-0d226d246990-kube-api-access-4t9d8\") pod \"maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:16.938298 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:16.938281    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6912507-9a44-4218-97ff-0d226d246990-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:16.989812 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:16.989783    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:17.039195 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:17.039173    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6912507-9a44-4218-97ff-0d226d246990-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:17.039317 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:17.039210    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:17.039317 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:17.039277    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:17.039403 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:17.039328    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9d8\" (UniqueName: \"kubernetes.io/projected/e6912507-9a44-4218-97ff-0d226d246990-kube-api-access-4t9d8\") pod \"maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:17.039448 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:02:17.039408    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-serving-cert: secret "maas-api-e2e-shared-b-w2-18b9d7-serving-cert" not found
Sep 12 01:02:17.039488 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:02:17.039474    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-maas-api-tls podName:e6912507-9a44-4218-97ff-0d226d246990 nodeName:}" failed. No retries permitted until 2026-09-12 01:02:17.539454677 +0000 UTC m=+1142.099906326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-maas-api-tls") pod "maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf" (UID: "e6912507-9a44-4218-97ff-0d226d246990") : secret "maas-api-e2e-shared-b-w2-18b9d7-serving-cert" not found
Sep 12 01:02:17.039819 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:17.039801    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6912507-9a44-4218-97ff-0d226d246990-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:17.041976 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:17.041938    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:17.051438 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:17.051409    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9d8\" (UniqueName: \"kubernetes.io/projected/e6912507-9a44-4218-97ff-0d226d246990-kube-api-access-4t9d8\") pod \"maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:17.201333 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:17.201299    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:17.415705 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:17.415676    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:17.516409 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:17.516299    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:17.544476 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:17.544433    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:17.547332 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:17.547301    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:17.809881 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:17.809789    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:17.997126 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:17.997097    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:18.089281 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:18.089254    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"]
Sep 12 01:02:18.091631 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:02:18.091605    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6912507_9a44_4218_97ff_0d226d246990.slice/crio-e4ac1153cbc53c058d1417cc27c10c5bd6890d8a7c0faa291594478699ada569 WatchSource:0}: Error finding container e4ac1153cbc53c058d1417cc27c10c5bd6890d8a7c0faa291594478699ada569: Status 404 returned error can't find the container with id e4ac1153cbc53c058d1417cc27c10c5bd6890d8a7c0faa291594478699ada569
Sep 12 01:02:18.519342 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:18.519317    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf" event={"ID":"e6912507-9a44-4218-97ff-0d226d246990","Type":"ContainerStarted","Data":"e4ac1153cbc53c058d1417cc27c10c5bd6890d8a7c0faa291594478699ada569"}
Sep 12 01:02:18.830605 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:18.830512    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:19.525768 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:19.525732    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf" event={"ID":"e6912507-9a44-4218-97ff-0d226d246990","Type":"ContainerStarted","Data":"02129ba15633fb586b54072dd7a824cabaa476ee913756cdad49a9170c1b151a"}
Sep 12 01:02:19.526163 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:19.525859    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:19.544527 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:19.544485    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf" podStartSLOduration=3.168025512 podStartE2EDuration="3.544471233s" totalImagesPullingTime="376.445721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:02:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:02:19.543821561 +0000 UTC m=+1144.104273242" watchObservedRunningTime="2026-09-12 01:02:19.544471233 +0000 UTC m=+1144.104922900"
Sep 12 01:02:19.674709 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:19.674678    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:20.502469 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:20.502438    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:21.198720 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:21.198687    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:21.885222 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:21.885190    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:21.984122 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:21.984091    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:22.145863 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:22.145774    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:22.789894 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:22.789836    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:23.591061 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:23.591002    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:24.478206 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:24.478172    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:25.164027 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:25.163996    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:25.535905 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:25.535835    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:02:25.995390 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:25.995354    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:26.681917 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:26.681884    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:27.467291 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:27.467259    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:28.219405 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:28.219373    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:29.320193 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:29.320164    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:29.890552 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:29.890500    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:30.620476 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:30.620439    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:31.595975 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:31.595935    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:32.404061 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:32.404022    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:32.662126 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:32.662095    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:32.995819 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:32.995737    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"]
Sep 12 01:02:32.996020 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:32.995995    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp" podUID="bbf87a86-35e2-449e-8e51-f93ca5c133d3" containerName="maas-api" containerID="cri-o://8ad6469033c2779b18554b0f586d923a2516b68ab4a77d27da030fe54a58aab7" gracePeriod=30
Sep 12 01:02:33.030649 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.030620    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:33.131771 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.131733    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:33.228450 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.228409    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:33.249344 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.249287    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 01:02:33.331741 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.331695    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:33.398222 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.398196    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qcqz\" (UniqueName: \"kubernetes.io/projected/bbf87a86-35e2-449e-8e51-f93ca5c133d3-kube-api-access-6qcqz\") pod \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") "
Sep 12 01:02:33.398360 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.398241    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf87a86-35e2-449e-8e51-f93ca5c133d3-maas-api-tls\") pod \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") "
Sep 12 01:02:33.398360 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.398314    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf87a86-35e2-449e-8e51-f93ca5c133d3-metrics-certs\") pod \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") "
Sep 12 01:02:33.398426 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.398411    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf87a86-35e2-449e-8e51-f93ca5c133d3-openshift-service-ca\") pod \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\" (UID: \"bbf87a86-35e2-449e-8e51-f93ca5c133d3\") "
Sep 12 01:02:33.398805 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.398776    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf87a86-35e2-449e-8e51-f93ca5c133d3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bbf87a86-35e2-449e-8e51-f93ca5c133d3" (UID: "bbf87a86-35e2-449e-8e51-f93ca5c133d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:02:33.400451 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.400426    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf87a86-35e2-449e-8e51-f93ca5c133d3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bbf87a86-35e2-449e-8e51-f93ca5c133d3" (UID: "bbf87a86-35e2-449e-8e51-f93ca5c133d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:02:33.400451 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.400433    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf87a86-35e2-449e-8e51-f93ca5c133d3-kube-api-access-6qcqz" (OuterVolumeSpecName: "kube-api-access-6qcqz") pod "bbf87a86-35e2-449e-8e51-f93ca5c133d3" (UID: "bbf87a86-35e2-449e-8e51-f93ca5c133d3"). InnerVolumeSpecName "kube-api-access-6qcqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:02:33.400591 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.400573    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf87a86-35e2-449e-8e51-f93ca5c133d3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bbf87a86-35e2-449e-8e51-f93ca5c133d3" (UID: "bbf87a86-35e2-449e-8e51-f93ca5c133d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:02:33.499638 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.499558    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qcqz\" (UniqueName: \"kubernetes.io/projected/bbf87a86-35e2-449e-8e51-f93ca5c133d3-kube-api-access-6qcqz\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:33.499638 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.499587    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf87a86-35e2-449e-8e51-f93ca5c133d3-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:33.499638 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.499599    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf87a86-35e2-449e-8e51-f93ca5c133d3-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:33.499638 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.499610    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf87a86-35e2-449e-8e51-f93ca5c133d3-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:33.598854 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.598822    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:33.602833 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.602808    2567 generic.go:356] "Generic (PLEG): container finished" podID="bbf87a86-35e2-449e-8e51-f93ca5c133d3" containerID="8ad6469033c2779b18554b0f586d923a2516b68ab4a77d27da030fe54a58aab7" exitCode=0
Sep 12 01:02:33.602995 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.602848    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp" event={"ID":"bbf87a86-35e2-449e-8e51-f93ca5c133d3","Type":"ContainerDied","Data":"8ad6469033c2779b18554b0f586d923a2516b68ab4a77d27da030fe54a58aab7"}
Sep 12 01:02:33.602995 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.602911    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp" event={"ID":"bbf87a86-35e2-449e-8e51-f93ca5c133d3","Type":"ContainerDied","Data":"1c3e1a224ab9710745aef63b6caee4632c6f4fe3dfa18484153d2598ed7694a2"}
Sep 12 01:02:33.602995 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.602858    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"
Sep 12 01:02:33.602995 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.602940    2567 scope.go:117] "RemoveContainer" containerID="8ad6469033c2779b18554b0f586d923a2516b68ab4a77d27da030fe54a58aab7"
Sep 12 01:02:33.612757 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.612741    2567 scope.go:117] "RemoveContainer" containerID="8ad6469033c2779b18554b0f586d923a2516b68ab4a77d27da030fe54a58aab7"
Sep 12 01:02:33.613053 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:02:33.613035    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad6469033c2779b18554b0f586d923a2516b68ab4a77d27da030fe54a58aab7\": container with ID starting with 8ad6469033c2779b18554b0f586d923a2516b68ab4a77d27da030fe54a58aab7 not found: ID does not exist" containerID="8ad6469033c2779b18554b0f586d923a2516b68ab4a77d27da030fe54a58aab7"
Sep 12 01:02:33.613108 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.613064    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad6469033c2779b18554b0f586d923a2516b68ab4a77d27da030fe54a58aab7"} err="failed to get container status \"8ad6469033c2779b18554b0f586d923a2516b68ab4a77d27da030fe54a58aab7\": rpc error: code = NotFound desc = could not find container \"8ad6469033c2779b18554b0f586d923a2516b68ab4a77d27da030fe54a58aab7\": container with ID starting with 8ad6469033c2779b18554b0f586d923a2516b68ab4a77d27da030fe54a58aab7 not found: ID does not exist"
Sep 12 01:02:33.623880 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.623859    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"]
Sep 12 01:02:33.627865 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.627845    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bf547a-8d5545fc9-9w5kp"]
Sep 12 01:02:33.711310 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.711281    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:33.974629 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:33.974595    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbf87a86-35e2-449e-8e51-f93ca5c133d3" path="/var/lib/kubelet/pods/bbf87a86-35e2-449e-8e51-f93ca5c133d3/volumes"
Sep 12 01:02:34.394103 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:34.394013    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:35.207284 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:35.207249    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:35.980817 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:35.980788    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:36.660632 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:36.660604    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:38.088530 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:38.088495    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:38.636613 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:38.636576    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:43.162462 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:43.162419    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:43.608787 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:43.608714    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:43.756633 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:43.756597    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:44.763723 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:44.763683    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:46.195236 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:46.195198    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:46.870076 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:46.870041    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:46.973975 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:46.973925    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:47.354377 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:47.354340    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:48.405329 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:48.405300    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:49.245193 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:49.245161    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:50.198208 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:50.198172    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:51.094049 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.093987    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"]
Sep 12 01:02:51.094626 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.094601    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbf87a86-35e2-449e-8e51-f93ca5c133d3" containerName="maas-api"
Sep 12 01:02:51.094626 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.094628    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbf87a86-35e2-449e-8e51-f93ca5c133d3" containerName="maas-api"
Sep 12 01:02:51.094822 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.094780    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbf87a86-35e2-449e-8e51-f93ca5c133d3" containerName="maas-api"
Sep 12 01:02:51.098075 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.098051    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:51.100209 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.100187    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0cfc70f5-serving-cert\""
Sep 12 01:02:51.110798 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.110778    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"]
Sep 12 01:02:51.125350 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.125328    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:51.167187 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.167151    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-maas-api-tls\") pod \"maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:51.167313 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.167206    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-openshift-service-ca\") pod \"maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:51.167313 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.167231    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-metrics-certs\") pod \"maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:51.167394 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.167371    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nhsb\" (UniqueName: \"kubernetes.io/projected/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-kube-api-access-6nhsb\") pod \"maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:51.188758 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.188729    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:51.268849 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.268815    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-maas-api-tls\") pod \"maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:51.268849 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.268853    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-openshift-service-ca\") pod \"maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:51.269383 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.268870    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-metrics-certs\") pod \"maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:51.269383 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.268935    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nhsb\" (UniqueName: \"kubernetes.io/projected/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-kube-api-access-6nhsb\") pod \"maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:51.269653 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.269615    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-openshift-service-ca\") pod \"maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:51.271479 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.271453    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-maas-api-tls\") pod \"maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:51.271643 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.271621    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-metrics-certs\") pod \"maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:51.277492 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.277463    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nhsb\" (UniqueName: \"kubernetes.io/projected/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-kube-api-access-6nhsb\") pod \"maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:51.290902 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.290880    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:51.409919 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.409887    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:51.635586 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.635543    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:51.690185 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.690161    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"]
Sep 12 01:02:51.692519 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:02:51.692491    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cf6945_dc2e_4118_80e2_2d45f46ba64d.slice/crio-6e70499e464df691a3fe72864f1da77a0f4636ea74c33dd7668199f22787c3ec WatchSource:0}: Error finding container 6e70499e464df691a3fe72864f1da77a0f4636ea74c33dd7668199f22787c3ec: Status 404 returned error can't find the container with id 6e70499e464df691a3fe72864f1da77a0f4636ea74c33dd7668199f22787c3ec
Sep 12 01:02:51.736370 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:51.736335    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:02:52.704820 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:52.704777    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5" event={"ID":"c3cf6945-dc2e-4118-80e2-2d45f46ba64d","Type":"ContainerStarted","Data":"38e406ea540e139e290283a2842ecdf624fd891d8e04d09a872fd131ae928b76"}
Sep 12 01:02:52.705253 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:52.704837    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5" event={"ID":"c3cf6945-dc2e-4118-80e2-2d45f46ba64d","Type":"ContainerStarted","Data":"6e70499e464df691a3fe72864f1da77a0f4636ea74c33dd7668199f22787c3ec"}
Sep 12 01:02:52.705253 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:52.704885    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:52.724745 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:52.724698    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5" podStartSLOduration=1.198735319 podStartE2EDuration="1.724683382s" totalImagesPullingTime="525.948063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:02:52.723279453 +0000 UTC m=+1177.283731132" watchObservedRunningTime="2026-09-12 01:02:52.724683382 +0000 UTC m=+1177.285135050"
Sep 12 01:02:58.715095 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:58.715059    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:02:59.747783 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:02:59.747735    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:12.929389 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:12.929349    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:13.306663 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:13.306564    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:14.099069 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:14.099038    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:14.767406 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:14.767377    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:15.596650 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:15.596619    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:15.933557 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:15.933531    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 01:03:15.935046 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:15.935025    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 01:03:16.080520 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:16.080494    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:16.296661 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:16.296573    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:16.381577 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:16.381546    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:16.941903 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:16.941871    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:17.049388 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:17.049355    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:17.304014 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:17.303915    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:17.400943 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:17.400905    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:17.632960 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:17.632873    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:19.461024 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:19.460991    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:20.209440 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:20.209407    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:20.298161 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:20.298118    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:21.099529 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:21.099493    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:21.790204 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:21.790177    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:22.464282 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:22.464245    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:23.302804 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:23.302693    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:23.964999 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:23.964967    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"]
Sep 12 01:03:23.968554 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:23.968533    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:23.971402 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:23.971380    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-61f73f-serving-cert\""
Sep 12 01:03:23.979004 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:23.978980    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"]
Sep 12 01:03:23.983482 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:23.983461    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/707d2410-20a8-49b8-ac59-553f5c12341d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:23.983578 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:23.983495    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:23.983578 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:23.983573    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdf2w\" (UniqueName: \"kubernetes.io/projected/707d2410-20a8-49b8-ac59-553f5c12341d-kube-api-access-mdf2w\") pod \"maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:23.983670 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:23.983591    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:23.995614 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:23.995590    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:24.084240 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.084207    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:24.084418 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.084293    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdf2w\" (UniqueName: \"kubernetes.io/projected/707d2410-20a8-49b8-ac59-553f5c12341d-kube-api-access-mdf2w\") pod \"maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:24.084418 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.084312    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:24.084418 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.084393    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/707d2410-20a8-49b8-ac59-553f5c12341d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:24.084588 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:03:24.084507    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-serving-cert: secret "maas-api-e2e-shared-a-w3-61f73f-serving-cert" not found
Sep 12 01:03:24.084588 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:03:24.084577    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-maas-api-tls podName:707d2410-20a8-49b8-ac59-553f5c12341d nodeName:}" failed. No retries permitted until 2026-09-12 01:03:24.584556796 +0000 UTC m=+1209.145008462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-maas-api-tls") pod "maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6" (UID: "707d2410-20a8-49b8-ac59-553f5c12341d") : secret "maas-api-e2e-shared-a-w3-61f73f-serving-cert" not found
Sep 12 01:03:24.085107 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.085088    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/707d2410-20a8-49b8-ac59-553f5c12341d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:24.086817 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.086795    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:24.093223 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.093201    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdf2w\" (UniqueName: \"kubernetes.io/projected/707d2410-20a8-49b8-ac59-553f5c12341d-kube-api-access-mdf2w\") pod \"maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:24.095896 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.095873    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:24.407476 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.407445    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:24.508836 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.508789    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:24.588155 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.588121    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:24.590798 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.590772    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:24.594752 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.594721    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:24.881756 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.881662    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:24.920427 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:24.920398    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:25.158127 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:25.158103    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"]
Sep 12 01:03:25.160547 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:03:25.160525    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707d2410_20a8_49b8_ac59_553f5c12341d.slice/crio-5a4a37d56689fc4c4ba2fb981667897cf25f78b10d19113468851f4f543d44f5 WatchSource:0}: Error finding container 5a4a37d56689fc4c4ba2fb981667897cf25f78b10d19113468851f4f543d44f5: Status 404 returned error can't find the container with id 5a4a37d56689fc4c4ba2fb981667897cf25f78b10d19113468851f4f543d44f5
Sep 12 01:03:25.880354 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:25.880268    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6" event={"ID":"707d2410-20a8-49b8-ac59-553f5c12341d","Type":"ContainerStarted","Data":"60200cd203815ea51828e8ee24fc3c9d5de93f9fa71f67f2f5f5642d7b054e87"}
Sep 12 01:03:25.880354 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:25.880350    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6" event={"ID":"707d2410-20a8-49b8-ac59-553f5c12341d","Type":"ContainerStarted","Data":"5a4a37d56689fc4c4ba2fb981667897cf25f78b10d19113468851f4f543d44f5"}
Sep 12 01:03:25.880531 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:25.880376    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:25.898694 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:25.898644    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6" podStartSLOduration=2.447119482 podStartE2EDuration="2.89863001s" totalImagesPullingTime="451.510528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:03:25.897031245 +0000 UTC m=+1210.457482932" watchObservedRunningTime="2026-09-12 01:03:25.89863001 +0000 UTC m=+1210.459081677"
Sep 12 01:03:31.890165 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:31.890136    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:03:39.185596 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:39.185558    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:40.277746 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:40.277715    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:46.064330 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:46.064289    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:46.657572 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:46.657542    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:46.891365 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:46.891335    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:47.613938 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:47.613903    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:48.480781 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:48.480745    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:49.188774 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:49.188731    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:49.953231 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:49.953192    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:50.244628 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:50.244540    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:50.577631 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:50.577554    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:50.825308 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:50.825276    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:50.904699 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:50.904620    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:51.103002 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:51.102965    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:53.788682 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:53.788646    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:54.490116 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:54.490079    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:54.592793 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:54.592758    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:55.483709 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:55.483671    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:55.855882 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:55.855773    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:56.054219 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:56.054186    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:56.165469 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:56.165442    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:56.301574 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:56.301547    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:57.170561 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:57.170529    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:57.890390 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:57.890356    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:58.577519 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:58.577489    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:59.373659 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.373625    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"]
Sep 12 01:03:59.378232 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.378210    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:03:59.380899 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.380876    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-ef5997-serving-cert\""
Sep 12 01:03:59.387765 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.387743    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"]
Sep 12 01:03:59.410154 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.410132    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:59.470544 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.470517    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:59.515184 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.515158    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:03:59.515295 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.515195    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bnpg\" (UniqueName: \"kubernetes.io/projected/c8f888b9-c82d-413f-8c22-0605cff89933-kube-api-access-7bnpg\") pod \"maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:03:59.515295 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.515220    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f888b9-c82d-413f-8c22-0605cff89933-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:03:59.515295 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.515261    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:03:59.616000 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.615931    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:03:59.616411 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.616024    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bnpg\" (UniqueName: \"kubernetes.io/projected/c8f888b9-c82d-413f-8c22-0605cff89933-kube-api-access-7bnpg\") pod \"maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:03:59.616411 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.616061    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f888b9-c82d-413f-8c22-0605cff89933-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:03:59.616411 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:03:59.616066    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-serving-cert: secret "maas-api-e2e-shared-b-w3-ef5997-serving-cert" not found
Sep 12 01:03:59.616411 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.616095    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:03:59.616411 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:03:59.616163    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-maas-api-tls podName:c8f888b9-c82d-413f-8c22-0605cff89933 nodeName:}" failed. No retries permitted until 2026-09-12 01:04:00.116139655 +0000 UTC m=+1244.676591306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-maas-api-tls") pod "maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n" (UID: "c8f888b9-c82d-413f-8c22-0605cff89933") : secret "maas-api-e2e-shared-b-w3-ef5997-serving-cert" not found
Sep 12 01:03:59.616788 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.616770    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f888b9-c82d-413f-8c22-0605cff89933-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:03:59.619146 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.619107    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:03:59.627689 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.627631    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bnpg\" (UniqueName: \"kubernetes.io/projected/c8f888b9-c82d-413f-8c22-0605cff89933-kube-api-access-7bnpg\") pod \"maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:03:59.705440 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.705412    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:59.897312 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.897211    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:03:59.996074 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:03:59.996040    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:00.122468 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:00.122419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:04:00.125005 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:00.124984    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:04:00.290669 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:00.290634    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:04:00.582083 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:00.582054    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"]
Sep 12 01:04:00.582985 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:04:00.582939    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f888b9_c82d_413f_8c22_0605cff89933.slice/crio-57b414e4ad927875cc5a2d4df42037f0f85fac4c60072dfefc979530061062f9 WatchSource:0}: Error finding container 57b414e4ad927875cc5a2d4df42037f0f85fac4c60072dfefc979530061062f9: Status 404 returned error can't find the container with id 57b414e4ad927875cc5a2d4df42037f0f85fac4c60072dfefc979530061062f9
Sep 12 01:04:01.055140 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:01.055107    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n" event={"ID":"c8f888b9-c82d-413f-8c22-0605cff89933","Type":"ContainerStarted","Data":"57b414e4ad927875cc5a2d4df42037f0f85fac4c60072dfefc979530061062f9"}
Sep 12 01:04:01.244420 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:01.244384    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:02.061815 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:02.061779    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n" event={"ID":"c8f888b9-c82d-413f-8c22-0605cff89933","Type":"ContainerStarted","Data":"ff1b474aa7e782bedb4b54b2c420cd0640831f7543174e5d3a8a30cf65dd47c6"}
Sep 12 01:04:02.062252 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:02.061921    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:04:02.081427 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:02.081375    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n" podStartSLOduration=2.6010078070000002 podStartE2EDuration="3.081357358s" totalImagesPullingTime="480.349551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:04:02.080086239 +0000 UTC m=+1246.640537918" watchObservedRunningTime="2026-09-12 01:04:02.081357358 +0000 UTC m=+1246.641809025"
Sep 12 01:04:08.072912 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:08.072834    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:04:09.163243 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:09.163213    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:21.558896 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:21.558865    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:23.469640 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:23.469606    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:23.918251 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:23.918221    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:24.065653 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:24.065622    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:24.554275 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:24.554241    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:24.789766 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:24.789733    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:25.494604 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:25.494572    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:26.197576 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:26.197532    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:26.898762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:26.898729    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:27.568123 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:27.568091    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:32.178447 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:32.178384    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:32.234558 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:32.234532    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:38.342550 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:38.342508    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:41.649074 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:41.649027    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:04:41.730280 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:04:41.730249    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:11.599116 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:11.599085    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:21.072684 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:21.072653    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:29.693960 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:29.693908    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:30.117832 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:30.117752    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"]
Sep 12 01:05:30.139195 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:30.139167    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:30.594298 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:30.594269    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:31.399915 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:31.399883    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:32.092162 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:32.092127    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:32.903928 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:32.903902    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:33.618407 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:33.618372    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:34.293697 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:34.293655    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:35.097300 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:35.097266    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:35.780236 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:35.780204    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:44.855221 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:44.855149    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"]
Sep 12 01:05:44.855606 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:44.855368    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5" podUID="c3cf6945-dc2e-4118-80e2-2d45f46ba64d" containerName="maas-api" containerID="cri-o://38e406ea540e139e290283a2842ecdf624fd891d8e04d09a872fd131ae928b76" gracePeriod=30
Sep 12 01:05:44.876152 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:44.876123    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:44.957041 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:44.957013    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:45.029389 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.029305    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:45.111366 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.111305    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:05:45.137753 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.137719    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f" podUID="b5245f7f-74f3-4ec5-bff4-663e759e7385" containerName="main" containerID="cri-o://1d94e69896c4f44470c7ccb19d8917df2f9950e32bf1ef98234b1b7ea394f69f" gracePeriod=45
Sep 12 01:05:45.162371 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.162344    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:45.227203 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.227171    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:45.272827 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.272792    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-maas-api-tls\") pod \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") "
Sep 12 01:05:45.273018 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.272864    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-metrics-certs\") pod \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") "
Sep 12 01:05:45.273018 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.272924    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nhsb\" (UniqueName: \"kubernetes.io/projected/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-kube-api-access-6nhsb\") pod \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") "
Sep 12 01:05:45.273152 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.273070    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-openshift-service-ca\") pod \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\" (UID: \"c3cf6945-dc2e-4118-80e2-2d45f46ba64d\") "
Sep 12 01:05:45.273562 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.273529    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3cf6945-dc2e-4118-80e2-2d45f46ba64d" (UID: "c3cf6945-dc2e-4118-80e2-2d45f46ba64d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:05:45.273767 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.273626    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:05:45.275282 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.275251    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c3cf6945-dc2e-4118-80e2-2d45f46ba64d" (UID: "c3cf6945-dc2e-4118-80e2-2d45f46ba64d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:05:45.275790 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.275755    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-kube-api-access-6nhsb" (OuterVolumeSpecName: "kube-api-access-6nhsb") pod "c3cf6945-dc2e-4118-80e2-2d45f46ba64d" (UID: "c3cf6945-dc2e-4118-80e2-2d45f46ba64d"). InnerVolumeSpecName "kube-api-access-6nhsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:05:45.275888 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.275864    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3cf6945-dc2e-4118-80e2-2d45f46ba64d" (UID: "c3cf6945-dc2e-4118-80e2-2d45f46ba64d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:05:45.374725 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.374641    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:05:45.374725 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.374677    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:05:45.374725 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.374689    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nhsb\" (UniqueName: \"kubernetes.io/projected/c3cf6945-dc2e-4118-80e2-2d45f46ba64d-kube-api-access-6nhsb\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:05:45.383887 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.383861    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:05:45.475177 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.475131    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5245f7f-74f3-4ec5-bff4-663e759e7385-tls-certs\") pod \"b5245f7f-74f3-4ec5-bff4-663e759e7385\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") "
Sep 12 01:05:45.475177 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.475192    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s625s\" (UniqueName: \"kubernetes.io/projected/b5245f7f-74f3-4ec5-bff4-663e759e7385-kube-api-access-s625s\") pod \"b5245f7f-74f3-4ec5-bff4-663e759e7385\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") "
Sep 12 01:05:45.475436 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.475228    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-tmp-dir\") pod \"b5245f7f-74f3-4ec5-bff4-663e759e7385\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") "
Sep 12 01:05:45.475436 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.475275    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-home\") pod \"b5245f7f-74f3-4ec5-bff4-663e759e7385\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") "
Sep 12 01:05:45.475436 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.475354    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-model-cache\") pod \"b5245f7f-74f3-4ec5-bff4-663e759e7385\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") "
Sep 12 01:05:45.475436 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.475394    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-dshm\") pod \"b5245f7f-74f3-4ec5-bff4-663e759e7385\" (UID: \"b5245f7f-74f3-4ec5-bff4-663e759e7385\") "
Sep 12 01:05:45.475658 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.475512    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5245f7f-74f3-4ec5-bff4-663e759e7385" (UID: "b5245f7f-74f3-4ec5-bff4-663e759e7385"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:05:45.475658 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.475561    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-home" (OuterVolumeSpecName: "home") pod "b5245f7f-74f3-4ec5-bff4-663e759e7385" (UID: "b5245f7f-74f3-4ec5-bff4-663e759e7385"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:05:45.475658 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.475612    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5245f7f-74f3-4ec5-bff4-663e759e7385" (UID: "b5245f7f-74f3-4ec5-bff4-663e759e7385"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:05:45.475844 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.475827    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-tmp-dir\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:05:45.475893 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.475846    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-home\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:05:45.475893 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.475854    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-model-cache\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:05:45.477552 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.477530    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5245f7f-74f3-4ec5-bff4-663e759e7385-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5245f7f-74f3-4ec5-bff4-663e759e7385" (UID: "b5245f7f-74f3-4ec5-bff4-663e759e7385"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:05:45.478087 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.478068    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-dshm" (OuterVolumeSpecName: "dshm") pod "b5245f7f-74f3-4ec5-bff4-663e759e7385" (UID: "b5245f7f-74f3-4ec5-bff4-663e759e7385"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:05:45.478167 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.478096    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5245f7f-74f3-4ec5-bff4-663e759e7385-kube-api-access-s625s" (OuterVolumeSpecName: "kube-api-access-s625s") pod "b5245f7f-74f3-4ec5-bff4-663e759e7385" (UID: "b5245f7f-74f3-4ec5-bff4-663e759e7385"). InnerVolumeSpecName "kube-api-access-s625s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:05:45.572213 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.572182    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:45.576915 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.576891    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s625s\" (UniqueName: \"kubernetes.io/projected/b5245f7f-74f3-4ec5-bff4-663e759e7385-kube-api-access-s625s\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:05:45.577067 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.576917    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5245f7f-74f3-4ec5-bff4-663e759e7385-dshm\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:05:45.577067 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.576932    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5245f7f-74f3-4ec5-bff4-663e759e7385-tls-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:05:45.585097 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.585070    2567 generic.go:356] "Generic (PLEG): container finished" podID="b5245f7f-74f3-4ec5-bff4-663e759e7385" containerID="1d94e69896c4f44470c7ccb19d8917df2f9950e32bf1ef98234b1b7ea394f69f" exitCode=0
Sep 12 01:05:45.585205 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.585115    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"
Sep 12 01:05:45.585205 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.585156    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f" event={"ID":"b5245f7f-74f3-4ec5-bff4-663e759e7385","Type":"ContainerDied","Data":"1d94e69896c4f44470c7ccb19d8917df2f9950e32bf1ef98234b1b7ea394f69f"}
Sep 12 01:05:45.585205 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.585201    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f" event={"ID":"b5245f7f-74f3-4ec5-bff4-663e759e7385","Type":"ContainerDied","Data":"8054ee297b7cb0484e305baf09ec9a0149d9d15e841fcc57dc37556db353802b"}
Sep 12 01:05:45.585329 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.585219    2567 scope.go:117] "RemoveContainer" containerID="1d94e69896c4f44470c7ccb19d8917df2f9950e32bf1ef98234b1b7ea394f69f"
Sep 12 01:05:45.586547 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.586520    2567 generic.go:356] "Generic (PLEG): container finished" podID="c3cf6945-dc2e-4118-80e2-2d45f46ba64d" containerID="38e406ea540e139e290283a2842ecdf624fd891d8e04d09a872fd131ae928b76" exitCode=0
Sep 12 01:05:45.586649 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.586553    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5" event={"ID":"c3cf6945-dc2e-4118-80e2-2d45f46ba64d","Type":"ContainerDied","Data":"38e406ea540e139e290283a2842ecdf624fd891d8e04d09a872fd131ae928b76"}
Sep 12 01:05:45.586649 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.586579    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"
Sep 12 01:05:45.586649 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.586593    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5" event={"ID":"c3cf6945-dc2e-4118-80e2-2d45f46ba64d","Type":"ContainerDied","Data":"6e70499e464df691a3fe72864f1da77a0f4636ea74c33dd7668199f22787c3ec"}
Sep 12 01:05:45.595830 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.595751    2567 scope.go:117] "RemoveContainer" containerID="1d94e69896c4f44470c7ccb19d8917df2f9950e32bf1ef98234b1b7ea394f69f"
Sep 12 01:05:45.596088 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:05:45.596065    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d94e69896c4f44470c7ccb19d8917df2f9950e32bf1ef98234b1b7ea394f69f\": container with ID starting with 1d94e69896c4f44470c7ccb19d8917df2f9950e32bf1ef98234b1b7ea394f69f not found: ID does not exist" containerID="1d94e69896c4f44470c7ccb19d8917df2f9950e32bf1ef98234b1b7ea394f69f"
Sep 12 01:05:45.596185 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.596096    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d94e69896c4f44470c7ccb19d8917df2f9950e32bf1ef98234b1b7ea394f69f"} err="failed to get container status \"1d94e69896c4f44470c7ccb19d8917df2f9950e32bf1ef98234b1b7ea394f69f\": rpc error: code = NotFound desc = could not find container \"1d94e69896c4f44470c7ccb19d8917df2f9950e32bf1ef98234b1b7ea394f69f\": container with ID starting with 1d94e69896c4f44470c7ccb19d8917df2f9950e32bf1ef98234b1b7ea394f69f not found: ID does not exist"
Sep 12 01:05:45.596185 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.596118    2567 scope.go:117] "RemoveContainer" containerID="38e406ea540e139e290283a2842ecdf624fd891d8e04d09a872fd131ae928b76"
Sep 12 01:05:45.604692 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.604673    2567 scope.go:117] "RemoveContainer" containerID="38e406ea540e139e290283a2842ecdf624fd891d8e04d09a872fd131ae928b76"
Sep 12 01:05:45.605280 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:05:45.605258    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e406ea540e139e290283a2842ecdf624fd891d8e04d09a872fd131ae928b76\": container with ID starting with 38e406ea540e139e290283a2842ecdf624fd891d8e04d09a872fd131ae928b76 not found: ID does not exist" containerID="38e406ea540e139e290283a2842ecdf624fd891d8e04d09a872fd131ae928b76"
Sep 12 01:05:45.605353 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.605286    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e406ea540e139e290283a2842ecdf624fd891d8e04d09a872fd131ae928b76"} err="failed to get container status \"38e406ea540e139e290283a2842ecdf624fd891d8e04d09a872fd131ae928b76\": rpc error: code = NotFound desc = could not find container \"38e406ea540e139e290283a2842ecdf624fd891d8e04d09a872fd131ae928b76\": container with ID starting with 38e406ea540e139e290283a2842ecdf624fd891d8e04d09a872fd131ae928b76 not found: ID does not exist"
Sep 12 01:05:45.607693 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.607672    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"]
Sep 12 01:05:45.611643 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.611624    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/e2e-conflict-a-f6b886e1-kserve-8675b996cf-q5z8f"]
Sep 12 01:05:45.621758 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.621739    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"]
Sep 12 01:05:45.626047 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.625989    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0cfc70f5-554b768987-rxmw5"]
Sep 12 01:05:45.973519 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.973481    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5245f7f-74f3-4ec5-bff4-663e759e7385" path="/var/lib/kubelet/pods/b5245f7f-74f3-4ec5-bff4-663e759e7385/volumes"
Sep 12 01:05:45.974113 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:45.974093    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3cf6945-dc2e-4118-80e2-2d45f46ba64d" path="/var/lib/kubelet/pods/c3cf6945-dc2e-4118-80e2-2d45f46ba64d/volumes"
Sep 12 01:05:46.311677 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:46.311583    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:47.150048 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.150021    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:47.421440 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.421411    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"]
Sep 12 01:05:47.421825 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.421812    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5245f7f-74f3-4ec5-bff4-663e759e7385" containerName="main"
Sep 12 01:05:47.421879 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.421827    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5245f7f-74f3-4ec5-bff4-663e759e7385" containerName="main"
Sep 12 01:05:47.421879 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.421849    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3cf6945-dc2e-4118-80e2-2d45f46ba64d" containerName="maas-api"
Sep 12 01:05:47.421879 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.421854    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3cf6945-dc2e-4118-80e2-2d45f46ba64d" containerName="maas-api"
Sep 12 01:05:47.421996 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.421915    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3cf6945-dc2e-4118-80e2-2d45f46ba64d" containerName="maas-api"
Sep 12 01:05:47.421996 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.421923    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5245f7f-74f3-4ec5-bff4-663e759e7385" containerName="main"
Sep 12 01:05:47.426602 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.426582    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.429161 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.429145    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f19fd1\"/\"distinct-w4-f19fd1-kserve-self-signed-certs\""
Sep 12 01:05:47.434567 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.434545    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"]
Sep 12 01:05:47.595689 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.595654    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-home\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.595855 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.595729    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6764424e-95bf-41a6-ae53-e6cb4b696731-tls-certs\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.595855 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.595761    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd86k\" (UniqueName: \"kubernetes.io/projected/6764424e-95bf-41a6-ae53-e6cb4b696731-kube-api-access-rd86k\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.595855 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.595804    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-dshm\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.595982 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.595902    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-model-cache\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.595982 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.595932    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-tmp-dir\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.697020 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.696904    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-home\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.697020 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.696985    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6764424e-95bf-41a6-ae53-e6cb4b696731-tls-certs\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.697020 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.697017    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd86k\" (UniqueName: \"kubernetes.io/projected/6764424e-95bf-41a6-ae53-e6cb4b696731-kube-api-access-rd86k\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.697307 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.697062    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-dshm\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.697401 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.697374    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-home\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.697555 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.697529    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-model-cache\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.697704 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.697574    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-tmp-dir\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.697968 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.697901    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-model-cache\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.698190 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.698169    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-tmp-dir\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.699774 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.699743    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-dshm\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.700145 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.700126    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6764424e-95bf-41a6-ae53-e6cb4b696731-tls-certs\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.709364 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.709339    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd86k\" (UniqueName: \"kubernetes.io/projected/6764424e-95bf-41a6-ae53-e6cb4b696731-kube-api-access-rd86k\") pod \"distinct-w4-f19fd1-kserve-765cd874f8-wv6v5\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") " pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.736680 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.736644    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:47.759273 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:47.759230    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:48.011150 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:48.011127    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"]
Sep 12 01:05:48.012765 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:05:48.012738    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6764424e_95bf_41a6_ae53_e6cb4b696731.slice/crio-881693f7f0eaaf8cac57e9af0b91a502af2f97e0830fc85015e0b7de4f7c1456 WatchSource:0}: Error finding container 881693f7f0eaaf8cac57e9af0b91a502af2f97e0830fc85015e0b7de4f7c1456: Status 404 returned error can't find the container with id 881693f7f0eaaf8cac57e9af0b91a502af2f97e0830fc85015e0b7de4f7c1456
Sep 12 01:05:48.164618 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:48.164590    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:48.441436 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:48.441406    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:48.605458 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:48.605417    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5" event={"ID":"6764424e-95bf-41a6-ae53-e6cb4b696731","Type":"ContainerStarted","Data":"859df7460f201d492fafe7670bfa57f9a68c9d758ce883af9c43322266def1e1"}
Sep 12 01:05:48.605458 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:48.605456    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5" event={"ID":"6764424e-95bf-41a6-ae53-e6cb4b696731","Type":"ContainerStarted","Data":"881693f7f0eaaf8cac57e9af0b91a502af2f97e0830fc85015e0b7de4f7c1456"}
Sep 12 01:05:48.625306 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:48.625258    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5" podStartSLOduration=1.625242805 podStartE2EDuration="1.625242805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:05:48.623043507 +0000 UTC m=+1353.183495177" watchObservedRunningTime="2026-09-12 01:05:48.625242805 +0000 UTC m=+1353.185694468"
Sep 12 01:05:49.003289 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:49.003256    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:49.691890 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:49.691857    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:50.547310 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:50.547279    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:53.514190 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:53.514159    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:56.022669 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:56.022617    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:05:57.737022 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:57.736994    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:57.737366 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:57.737037    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:57.749403 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:57.749383    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:05:58.671848 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:05:58.671820    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:06:30.669590 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:30.669557    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:30.993847 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:30.993761    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:36.672730 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:36.672700    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:37.551865 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:37.551830    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:37.680345 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:37.680309    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:39.020534 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:39.020502    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:42.387289 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:42.387258    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:42.456181 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:42.456152    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:42.671981 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:42.671932    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:43.931044 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:43.930978    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:44.024352 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:44.024317    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:44.124859 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:44.124826    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:44.734382 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:44.734345    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:44.835914 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:44.835878    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:44.936217 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:44.936184    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:46.217500 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:46.217458    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:52.545682 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:52.545649    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:52.646270 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:52.646236    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:53.228223 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:53.228187    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:59.165286 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:59.165232    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:59.273531 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:59.273498    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:06:59.364750 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:06:59.364723    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:06.991329 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:06.991239    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:09.850965 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:09.850909    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:10.069744 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:10.069694    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:16.543997 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:16.543966    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:17.184356 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:17.184324    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:31.017621 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:31.017583    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:34.427381 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:34.427344    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:34.585572 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:34.585541    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:38.258745 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:38.258712    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:38.358347 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:38.358313    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:38.630586 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:38.630499    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:38.729560 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:38.729531    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:38.985957 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:38.985925    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:39.156240 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:39.156208    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:41.999329 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:41.999291    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:42.168782 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:42.168738    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:42.264208 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:42.264133    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:43.029676 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:43.029646    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:43.747081 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:43.747052    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:44.220685 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.220653    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"]
Sep 12 01:07:44.224269 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.224251    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.228051 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.228021    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10ba6f\"/\"embedding-w0-10ba6f-kserve-self-signed-certs\""
Sep 12 01:07:44.236082 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.236058    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"]
Sep 12 01:07:44.263282 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.263248    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-model-cache\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.263414 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.263312    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-dshm\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.263414 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.263340    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-tmp-dir\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.263503 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.263447    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-home\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.263503 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.263493    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-tls-certs\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.263577 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.263554    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x7fs\" (UniqueName: \"kubernetes.io/projected/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-kube-api-access-7x7fs\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.325616 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.325586    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:44.364971 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.364916    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-home\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.365113 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.365005    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-tls-certs\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.365113 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.365083    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x7fs\" (UniqueName: \"kubernetes.io/projected/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-kube-api-access-7x7fs\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.365228 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.365123    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-model-cache\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.365228 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.365191    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-dshm\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.365228 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.365219    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-tmp-dir\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.365383 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.365361    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-home\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.365620 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.365591    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-tmp-dir\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.365816 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.365590    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-model-cache\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.367702 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.367679    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-dshm\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.367992 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.367974    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-tls-certs\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.375034 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.375011    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x7fs\" (UniqueName: \"kubernetes.io/projected/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-kube-api-access-7x7fs\") pod \"embedding-w0-10ba6f-kserve-54786fb55b-cpcb8\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") " pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.533998 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.533903    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:44.702179 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.702146    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:44.805015 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:44.804990    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"]
Sep 12 01:07:44.806782 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:07:44.806740    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dd44c11_5a0e_4a4a_8a9c_9c45ce4b7a18.slice/crio-93ae6a3bbebe4a5fa261d28b5e054d4bb9859a8e1621f8654636d4c6052f0486 WatchSource:0}: Error finding container 93ae6a3bbebe4a5fa261d28b5e054d4bb9859a8e1621f8654636d4c6052f0486: Status 404 returned error can't find the container with id 93ae6a3bbebe4a5fa261d28b5e054d4bb9859a8e1621f8654636d4c6052f0486
Sep 12 01:07:45.221164 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:45.221122    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8" event={"ID":"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18","Type":"ContainerStarted","Data":"4b40003a2ba9320a567ed5fd9087c6975fe5679fd0e1da6e0b5f1f77b40e125d"}
Sep 12 01:07:45.221551 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:45.221170    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8" event={"ID":"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18","Type":"ContainerStarted","Data":"93ae6a3bbebe4a5fa261d28b5e054d4bb9859a8e1621f8654636d4c6052f0486"}
Sep 12 01:07:45.240913 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:45.240864    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8" podStartSLOduration=1.240851086 podStartE2EDuration="1.240851086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:07:45.238997539 +0000 UTC m=+1469.799449218" watchObservedRunningTime="2026-09-12 01:07:45.240851086 +0000 UTC m=+1469.801302754"
Sep 12 01:07:45.374624 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:45.374592    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:46.290656 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:46.290626    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:46.967642 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:46.967612    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"]
Sep 12 01:07:46.971476 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:46.971452    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:46.974033 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:46.974003    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-033ca6-serving-cert\""
Sep 12 01:07:46.982134 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:46.982107    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"]
Sep 12 01:07:46.988710 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:46.988674    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-metrics-certs\") pod \"maas-api-e2e-api-a-033ca6-7974859fcd-v757q\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:46.988839 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:46.988821    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79bzv\" (UniqueName: \"kubernetes.io/projected/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-kube-api-access-79bzv\") pod \"maas-api-e2e-api-a-033ca6-7974859fcd-v757q\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:46.988908 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:46.988894    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-maas-api-tls\") pod \"maas-api-e2e-api-a-033ca6-7974859fcd-v757q\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:46.989019 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:46.988998    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-openshift-service-ca\") pod \"maas-api-e2e-api-a-033ca6-7974859fcd-v757q\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:47.002973 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.002918    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:47.089922 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.089885    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79bzv\" (UniqueName: \"kubernetes.io/projected/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-kube-api-access-79bzv\") pod \"maas-api-e2e-api-a-033ca6-7974859fcd-v757q\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:47.090131 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.089979    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-maas-api-tls\") pod \"maas-api-e2e-api-a-033ca6-7974859fcd-v757q\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:47.090131 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.090030    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-openshift-service-ca\") pod \"maas-api-e2e-api-a-033ca6-7974859fcd-v757q\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:47.090131 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.090085    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-metrics-certs\") pod \"maas-api-e2e-api-a-033ca6-7974859fcd-v757q\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:47.090710 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.090684    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-openshift-service-ca\") pod \"maas-api-e2e-api-a-033ca6-7974859fcd-v757q\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:47.092526 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.092503    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-maas-api-tls\") pod \"maas-api-e2e-api-a-033ca6-7974859fcd-v757q\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:47.092637 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.092562    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-metrics-certs\") pod \"maas-api-e2e-api-a-033ca6-7974859fcd-v757q\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:47.099336 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.099311    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79bzv\" (UniqueName: \"kubernetes.io/projected/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-kube-api-access-79bzv\") pod \"maas-api-e2e-api-a-033ca6-7974859fcd-v757q\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:47.105121 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.105096    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:47.203785 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.203750    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:47.286282 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.286190    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:47.496851 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.496820    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:47.563195 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.563168    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"]
Sep 12 01:07:47.564965 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.564920    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:07:47.565282 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:07:47.565238    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f35eb0_2d56_4141_b204_8cf8f0d35dbd.slice/crio-7a4bd286844bb77d976e947df8176bb6bbc741265728be09a5e51c9bacbb37f6 WatchSource:0}: Error finding container 7a4bd286844bb77d976e947df8176bb6bbc741265728be09a5e51c9bacbb37f6: Status 404 returned error can't find the container with id 7a4bd286844bb77d976e947df8176bb6bbc741265728be09a5e51c9bacbb37f6
Sep 12 01:07:47.566911 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:47.566889    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 01:07:48.240369 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:48.240337    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q" event={"ID":"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd","Type":"ContainerStarted","Data":"05d5790988d5421a34892ae43dab7d7aec0d5d04b94a18b854711a48340c8e43"}
Sep 12 01:07:48.240552 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:48.240392    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q" event={"ID":"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd","Type":"ContainerStarted","Data":"7a4bd286844bb77d976e947df8176bb6bbc741265728be09a5e51c9bacbb37f6"}
Sep 12 01:07:48.240552 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:48.240423    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:48.279571 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:48.279479    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q" podStartSLOduration=1.824783235 podStartE2EDuration="2.279461952s" totalImagesPullingTime="454.678717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:07:48.277923938 +0000 UTC m=+1472.838375620" watchObservedRunningTime="2026-09-12 01:07:48.279461952 +0000 UTC m=+1472.839913620"
Sep 12 01:07:54.250359 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:54.250327    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:07:54.534526 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:54.534438    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:54.534526 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:54.534484    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:54.546647 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:54.546611    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:07:55.300170 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:07:55.300141    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:08:08.974865 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:08.974830    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:09.753305 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:09.753268    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:10.503924 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:10.503892    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:11.187782 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:11.187751    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:12.019629 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:12.019594    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:12.262318 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:12.262284    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:12.801077 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:12.801034    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:12.971807 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:12.971778    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:13.157181 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:13.157151    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:13.279736 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:13.279711    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:13.792693 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:13.792661    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:14.484678 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:14.484648    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:15.712299 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:15.712260    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:15.970645 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:15.970562    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 01:08:15.972574 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:15.972553    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 01:08:16.392156 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:16.392081    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:16.487851 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:16.487819    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:16.725046 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:16.724942    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:17.592567 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:17.592534    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:18.434701 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:18.434668    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:19.298747 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:19.298707    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:20.006258 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:20.006224    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:20.907911 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:20.907880    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:21.689626 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:21.689593    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:22.458825 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.458796    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"]
Sep 12 01:08:22.462913 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.462891    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:22.465351 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.465327    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-49980f-serving-cert\""
Sep 12 01:08:22.472302 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.471652    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"]
Sep 12 01:08:22.494486 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.494461    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:22.557023 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.556992    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d57b\" (UniqueName: \"kubernetes.io/projected/a85529a6-3f04-498a-a8a7-da73662c5f67-kube-api-access-9d57b\") pod \"maas-api-e2e-api-b-49980f-bff7f6c9f-skznc\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:22.557185 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.557040    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85529a6-3f04-498a-a8a7-da73662c5f67-openshift-service-ca\") pod \"maas-api-e2e-api-b-49980f-bff7f6c9f-skznc\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:22.557185 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.557102    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-metrics-certs\") pod \"maas-api-e2e-api-b-49980f-bff7f6c9f-skznc\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:22.557287 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.557187    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-maas-api-tls\") pod \"maas-api-e2e-api-b-49980f-bff7f6c9f-skznc\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:22.562918 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.562892    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:22.658189 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.658149    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-maas-api-tls\") pod \"maas-api-e2e-api-b-49980f-bff7f6c9f-skznc\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:22.658365 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.658251    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d57b\" (UniqueName: \"kubernetes.io/projected/a85529a6-3f04-498a-a8a7-da73662c5f67-kube-api-access-9d57b\") pod \"maas-api-e2e-api-b-49980f-bff7f6c9f-skznc\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:22.658365 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:08:22.658296    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-serving-cert: secret "maas-api-e2e-api-b-49980f-serving-cert" not found
Sep 12 01:08:22.658365 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:08:22.658364    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-maas-api-tls podName:a85529a6-3f04-498a-a8a7-da73662c5f67 nodeName:}" failed. No retries permitted until 2026-09-12 01:08:23.158347992 +0000 UTC m=+1507.718799639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-maas-api-tls") pod "maas-api-e2e-api-b-49980f-bff7f6c9f-skznc" (UID: "a85529a6-3f04-498a-a8a7-da73662c5f67") : secret "maas-api-e2e-api-b-49980f-serving-cert" not found
Sep 12 01:08:22.658560 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.658300    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85529a6-3f04-498a-a8a7-da73662c5f67-openshift-service-ca\") pod \"maas-api-e2e-api-b-49980f-bff7f6c9f-skznc\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:22.658560 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.658436    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-metrics-certs\") pod \"maas-api-e2e-api-b-49980f-bff7f6c9f-skznc\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:22.658941 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.658918    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85529a6-3f04-498a-a8a7-da73662c5f67-openshift-service-ca\") pod \"maas-api-e2e-api-b-49980f-bff7f6c9f-skznc\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:22.661169 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.661145    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-metrics-certs\") pod \"maas-api-e2e-api-b-49980f-bff7f6c9f-skznc\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:22.669130 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.669106    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d57b\" (UniqueName: \"kubernetes.io/projected/a85529a6-3f04-498a-a8a7-da73662c5f67-kube-api-access-9d57b\") pod \"maas-api-e2e-api-b-49980f-bff7f6c9f-skznc\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:22.705187 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:22.705127    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:23.079652 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:23.079585    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:23.164630 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:23.164597    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-maas-api-tls\") pod \"maas-api-e2e-api-b-49980f-bff7f6c9f-skznc\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:23.167161 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:23.167138    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-maas-api-tls\") pod \"maas-api-e2e-api-b-49980f-bff7f6c9f-skznc\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:23.176596 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:23.176571    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:23.377790 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:23.377633    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:23.504887 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:23.504837    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:23.649682 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:23.649584    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"]
Sep 12 01:08:23.652191 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:08:23.652158    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85529a6_3f04_498a_a8a7_da73662c5f67.slice/crio-a03bcef1444cb89afaf2b95fc079d33646976a3374d75d23aeed4f278021b252 WatchSource:0}: Error finding container a03bcef1444cb89afaf2b95fc079d33646976a3374d75d23aeed4f278021b252: Status 404 returned error can't find the container with id a03bcef1444cb89afaf2b95fc079d33646976a3374d75d23aeed4f278021b252
Sep 12 01:08:24.286684 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:24.286611    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:24.436190 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:24.436158    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc" event={"ID":"a85529a6-3f04-498a-a8a7-da73662c5f67","Type":"ContainerStarted","Data":"c6d805cffe0aa657c22ba99fc23731428277cb200a3c688151ebbf581b86f37a"}
Sep 12 01:08:24.436190 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:24.436207    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc" event={"ID":"a85529a6-3f04-498a-a8a7-da73662c5f67","Type":"ContainerStarted","Data":"a03bcef1444cb89afaf2b95fc079d33646976a3374d75d23aeed4f278021b252"}
Sep 12 01:08:24.436436 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:24.436309    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:24.455473 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:24.455422    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc" podStartSLOduration=2.154088225 podStartE2EDuration="2.455408876s" totalImagesPullingTime="301.320651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:08:24.454459895 +0000 UTC m=+1509.014911562" watchObservedRunningTime="2026-09-12 01:08:24.455408876 +0000 UTC m=+1509.015860545"
Sep 12 01:08:30.447203 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:30.447177    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:08:44.909110 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:44.909032    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:45.621096 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:45.621060    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:46.298346 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:46.298317    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:46.444601 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:46.444571    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:47.177971 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:47.177917    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:47.925368 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:47.925336    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:48.863829 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:48.863795    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:49.824653 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:49.824624    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:50.308573 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:50.308542    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:50.683970 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:50.683920    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:51.652266 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:51.652228    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:52.377986 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:52.377934    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:53.790440 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:53.790406    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:53.891933 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:53.891902    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:54.681002 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:54.680973    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:55.357894 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:55.357865    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:08:56.023778 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:08:56.023747    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:03.197927 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:03.197890    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:03.253032 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:03.253006    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:04.928011 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:04.927979    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"]
Sep 12 01:09:04.928484 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:04.928221    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q" podUID="c2f35eb0-2d56-4141-b204-8cf8f0d35dbd" containerName="maas-api" containerID="cri-o://05d5790988d5421a34892ae43dab7d7aec0d5d04b94a18b854711a48340c8e43" gracePeriod=30
Sep 12 01:09:04.961195 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:04.961166    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:05.030815 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.030770    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:05.096022 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.095992    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:05.165472 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.165445    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:05.182752 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.182694    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:09:05.234460 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.234431    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:05.273888 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.273867    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-maas-api-tls\") pod \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") "
Sep 12 01:09:05.274023 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.273936    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-openshift-service-ca\") pod \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") "
Sep 12 01:09:05.274072 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.274048    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79bzv\" (UniqueName: \"kubernetes.io/projected/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-kube-api-access-79bzv\") pod \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") "
Sep 12 01:09:05.274128 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.274095    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-metrics-certs\") pod \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\" (UID: \"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd\") "
Sep 12 01:09:05.274354 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.274322    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2f35eb0-2d56-4141-b204-8cf8f0d35dbd" (UID: "c2f35eb0-2d56-4141-b204-8cf8f0d35dbd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:05.274493 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.274475    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:05.276245 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.276222    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c2f35eb0-2d56-4141-b204-8cf8f0d35dbd" (UID: "c2f35eb0-2d56-4141-b204-8cf8f0d35dbd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:09:05.276679 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.276660    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-kube-api-access-79bzv" (OuterVolumeSpecName: "kube-api-access-79bzv") pod "c2f35eb0-2d56-4141-b204-8cf8f0d35dbd" (UID: "c2f35eb0-2d56-4141-b204-8cf8f0d35dbd"). InnerVolumeSpecName "kube-api-access-79bzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:09:05.276741 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.276667    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2f35eb0-2d56-4141-b204-8cf8f0d35dbd" (UID: "c2f35eb0-2d56-4141-b204-8cf8f0d35dbd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:09:05.375035 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.375002    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79bzv\" (UniqueName: \"kubernetes.io/projected/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-kube-api-access-79bzv\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:05.375035 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.375030    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:05.375035 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.375039    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:05.541336 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.541242    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:05.641832 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.641797    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:05.653396 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.653369    2567 generic.go:356] "Generic (PLEG): container finished" podID="c2f35eb0-2d56-4141-b204-8cf8f0d35dbd" containerID="05d5790988d5421a34892ae43dab7d7aec0d5d04b94a18b854711a48340c8e43" exitCode=0
Sep 12 01:09:05.653537 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.653417    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"
Sep 12 01:09:05.653537 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.653449    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q" event={"ID":"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd","Type":"ContainerDied","Data":"05d5790988d5421a34892ae43dab7d7aec0d5d04b94a18b854711a48340c8e43"}
Sep 12 01:09:05.653537 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.653499    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q" event={"ID":"c2f35eb0-2d56-4141-b204-8cf8f0d35dbd","Type":"ContainerDied","Data":"7a4bd286844bb77d976e947df8176bb6bbc741265728be09a5e51c9bacbb37f6"}
Sep 12 01:09:05.653537 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.653521    2567 scope.go:117] "RemoveContainer" containerID="05d5790988d5421a34892ae43dab7d7aec0d5d04b94a18b854711a48340c8e43"
Sep 12 01:09:05.665847 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.665828    2567 scope.go:117] "RemoveContainer" containerID="05d5790988d5421a34892ae43dab7d7aec0d5d04b94a18b854711a48340c8e43"
Sep 12 01:09:05.666149 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:09:05.666130    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d5790988d5421a34892ae43dab7d7aec0d5d04b94a18b854711a48340c8e43\": container with ID starting with 05d5790988d5421a34892ae43dab7d7aec0d5d04b94a18b854711a48340c8e43 not found: ID does not exist" containerID="05d5790988d5421a34892ae43dab7d7aec0d5d04b94a18b854711a48340c8e43"
Sep 12 01:09:05.666226 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.666162    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d5790988d5421a34892ae43dab7d7aec0d5d04b94a18b854711a48340c8e43"} err="failed to get container status \"05d5790988d5421a34892ae43dab7d7aec0d5d04b94a18b854711a48340c8e43\": rpc error: code = NotFound desc = could not find container \"05d5790988d5421a34892ae43dab7d7aec0d5d04b94a18b854711a48340c8e43\": container with ID starting with 05d5790988d5421a34892ae43dab7d7aec0d5d04b94a18b854711a48340c8e43 not found: ID does not exist"
Sep 12 01:09:05.678759 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.678732    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"]
Sep 12 01:09:05.683032 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.683014    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-033ca6-7974859fcd-v757q"]
Sep 12 01:09:05.973939 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:05.973906    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2f35eb0-2d56-4141-b204-8cf8f0d35dbd" path="/var/lib/kubelet/pods/c2f35eb0-2d56-4141-b204-8cf8f0d35dbd/volumes"
Sep 12 01:09:06.321423 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:06.321351    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:07.263246 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:07.263215    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:08.089793 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:08.089758    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:08.959885 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:08.959850    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:09.708796 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:09.708768    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:10.616992 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:10.616940    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:11.357102 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:11.357069    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:12.068666 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:12.068631    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:12.128224 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:12.128191    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:12.358460 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:12.358373    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:13.104837 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:13.104806    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:13.790686 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:13.790656    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:14.485305 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:14.485274    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:15.183340 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:15.183302    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:16.181518 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:16.181482    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:16.880278 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:16.880235    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:17.717530 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:17.717503    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:18.406074 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:18.406023    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:19.105854 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:19.105825    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:20.906059 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:20.906021    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:22.811634 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:22.811550    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:28.416886 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.416845    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"]
Sep 12 01:09:28.417379 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.417077    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc" podUID="a85529a6-3f04-498a-a8a7-da73662c5f67" containerName="maas-api" containerID="cri-o://c6d805cffe0aa657c22ba99fc23731428277cb200a3c688151ebbf581b86f37a" gracePeriod=30
Sep 12 01:09:28.445159 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.445136    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:28.668275 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.668217    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:09:28.713635 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.713607    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-maas-api-tls\") pod \"a85529a6-3f04-498a-a8a7-da73662c5f67\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") "
Sep 12 01:09:28.713820 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.713683    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85529a6-3f04-498a-a8a7-da73662c5f67-openshift-service-ca\") pod \"a85529a6-3f04-498a-a8a7-da73662c5f67\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") "
Sep 12 01:09:28.713820 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.713721    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-metrics-certs\") pod \"a85529a6-3f04-498a-a8a7-da73662c5f67\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") "
Sep 12 01:09:28.713820 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.713802    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d57b\" (UniqueName: \"kubernetes.io/projected/a85529a6-3f04-498a-a8a7-da73662c5f67-kube-api-access-9d57b\") pod \"a85529a6-3f04-498a-a8a7-da73662c5f67\" (UID: \"a85529a6-3f04-498a-a8a7-da73662c5f67\") "
Sep 12 01:09:28.714134 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.714104    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85529a6-3f04-498a-a8a7-da73662c5f67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a85529a6-3f04-498a-a8a7-da73662c5f67" (UID: "a85529a6-3f04-498a-a8a7-da73662c5f67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:28.714245 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.714215    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85529a6-3f04-498a-a8a7-da73662c5f67-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:28.716030 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.716002    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a85529a6-3f04-498a-a8a7-da73662c5f67" (UID: "a85529a6-3f04-498a-a8a7-da73662c5f67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:09:28.716446 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.716420    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85529a6-3f04-498a-a8a7-da73662c5f67-kube-api-access-9d57b" (OuterVolumeSpecName: "kube-api-access-9d57b") pod "a85529a6-3f04-498a-a8a7-da73662c5f67" (UID: "a85529a6-3f04-498a-a8a7-da73662c5f67"). InnerVolumeSpecName "kube-api-access-9d57b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:09:28.716560 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.716492    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a85529a6-3f04-498a-a8a7-da73662c5f67" (UID: "a85529a6-3f04-498a-a8a7-da73662c5f67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:09:28.774397 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.774368    2567 generic.go:356] "Generic (PLEG): container finished" podID="a85529a6-3f04-498a-a8a7-da73662c5f67" containerID="c6d805cffe0aa657c22ba99fc23731428277cb200a3c688151ebbf581b86f37a" exitCode=0
Sep 12 01:09:28.774502 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.774411    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"
Sep 12 01:09:28.774502 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.774442    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc" event={"ID":"a85529a6-3f04-498a-a8a7-da73662c5f67","Type":"ContainerDied","Data":"c6d805cffe0aa657c22ba99fc23731428277cb200a3c688151ebbf581b86f37a"}
Sep 12 01:09:28.774502 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.774493    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc" event={"ID":"a85529a6-3f04-498a-a8a7-da73662c5f67","Type":"ContainerDied","Data":"a03bcef1444cb89afaf2b95fc079d33646976a3374d75d23aeed4f278021b252"}
Sep 12 01:09:28.774622 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.774515    2567 scope.go:117] "RemoveContainer" containerID="c6d805cffe0aa657c22ba99fc23731428277cb200a3c688151ebbf581b86f37a"
Sep 12 01:09:28.784471 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.784452    2567 scope.go:117] "RemoveContainer" containerID="c6d805cffe0aa657c22ba99fc23731428277cb200a3c688151ebbf581b86f37a"
Sep 12 01:09:28.784827 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:09:28.784807    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d805cffe0aa657c22ba99fc23731428277cb200a3c688151ebbf581b86f37a\": container with ID starting with c6d805cffe0aa657c22ba99fc23731428277cb200a3c688151ebbf581b86f37a not found: ID does not exist" containerID="c6d805cffe0aa657c22ba99fc23731428277cb200a3c688151ebbf581b86f37a"
Sep 12 01:09:28.784890 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.784844    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d805cffe0aa657c22ba99fc23731428277cb200a3c688151ebbf581b86f37a"} err="failed to get container status \"c6d805cffe0aa657c22ba99fc23731428277cb200a3c688151ebbf581b86f37a\": rpc error: code = NotFound desc = could not find container \"c6d805cffe0aa657c22ba99fc23731428277cb200a3c688151ebbf581b86f37a\": container with ID starting with c6d805cffe0aa657c22ba99fc23731428277cb200a3c688151ebbf581b86f37a not found: ID does not exist"
Sep 12 01:09:28.799150 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.799125    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"]
Sep 12 01:09:28.801119 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.801100    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49980f-bff7f6c9f-skznc"]
Sep 12 01:09:28.815075 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.815053    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d57b\" (UniqueName: \"kubernetes.io/projected/a85529a6-3f04-498a-a8a7-da73662c5f67-kube-api-access-9d57b\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:28.815173 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.815077    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:28.815173 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:28.815092    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85529a6-3f04-498a-a8a7-da73662c5f67-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:29.930552 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:29.930518    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:29.974380 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:29.974351    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a85529a6-3f04-498a-a8a7-da73662c5f67" path="/var/lib/kubelet/pods/a85529a6-3f04-498a-a8a7-da73662c5f67/volumes"
Sep 12 01:09:29.993566 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:29.993538    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:30.713685 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:30.713656    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:31.565739 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:31.565687    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:32.237520 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:32.237482    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:33.002047 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:33.002012    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:33.683286 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:33.683256    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:34.387264 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:34.387229    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"]
Sep 12 01:09:34.396877 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:34.396847    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"]
Sep 12 01:09:34.466438 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:34.466407    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:34.615146 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:34.615104    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:35.286928 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:35.286898    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:35.358994 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:35.358939    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:35.555880 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:35.555802    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:36.005568 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:36.005535    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:36.986343 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:36.986308    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:37.890919 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:37.890886    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:38.684043 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:38.684007    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:39.385290 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:39.385245    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:40.283189 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:40.283136    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:41.079509 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:41.079460    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:45.053596 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.053567    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"]
Sep 12 01:09:45.054018 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.053821    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n" podUID="e8a35c6d-185e-485a-bece-6da769a0f541" containerName="maas-api" containerID="cri-o://fb4c9c01f7666c5cfe76ea6441792a091554e3271e868504d534ffeeb6ab3267" gracePeriod=30
Sep 12 01:09:45.078823 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.078765    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:45.179919 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.179824    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:45.276100 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.276065    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:45.303677 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.303579    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 01:09:45.374690 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.374654    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-maas-api-tls\") pod \"e8a35c6d-185e-485a-bece-6da769a0f541\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") "
Sep 12 01:09:45.374864 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.374742    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49dnl\" (UniqueName: \"kubernetes.io/projected/e8a35c6d-185e-485a-bece-6da769a0f541-kube-api-access-49dnl\") pod \"e8a35c6d-185e-485a-bece-6da769a0f541\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") "
Sep 12 01:09:45.374864 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.374828    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a35c6d-185e-485a-bece-6da769a0f541-openshift-service-ca\") pod \"e8a35c6d-185e-485a-bece-6da769a0f541\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") "
Sep 12 01:09:45.375035 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.374974    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-metrics-certs\") pod \"e8a35c6d-185e-485a-bece-6da769a0f541\" (UID: \"e8a35c6d-185e-485a-bece-6da769a0f541\") "
Sep 12 01:09:45.375214 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.375189    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a35c6d-185e-485a-bece-6da769a0f541-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e8a35c6d-185e-485a-bece-6da769a0f541" (UID: "e8a35c6d-185e-485a-bece-6da769a0f541"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:45.375368 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.375352    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a35c6d-185e-485a-bece-6da769a0f541-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:45.377146 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.377114    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e8a35c6d-185e-485a-bece-6da769a0f541" (UID: "e8a35c6d-185e-485a-bece-6da769a0f541"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:09:45.377263 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.377227    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:45.377332 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.377294    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e8a35c6d-185e-485a-bece-6da769a0f541" (UID: "e8a35c6d-185e-485a-bece-6da769a0f541"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:09:45.377457 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.377439    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a35c6d-185e-485a-bece-6da769a0f541-kube-api-access-49dnl" (OuterVolumeSpecName: "kube-api-access-49dnl") pod "e8a35c6d-185e-485a-bece-6da769a0f541" (UID: "e8a35c6d-185e-485a-bece-6da769a0f541"). InnerVolumeSpecName "kube-api-access-49dnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:09:45.476730 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.476701    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:45.476730 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.476731    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a35c6d-185e-485a-bece-6da769a0f541-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:45.476908 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.476740    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49dnl\" (UniqueName: \"kubernetes.io/projected/e8a35c6d-185e-485a-bece-6da769a0f541-kube-api-access-49dnl\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:45.665299 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.665273    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:45.789144 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.789115    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:45.861133 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.861099    2567 generic.go:356] "Generic (PLEG): container finished" podID="e8a35c6d-185e-485a-bece-6da769a0f541" containerID="fb4c9c01f7666c5cfe76ea6441792a091554e3271e868504d534ffeeb6ab3267" exitCode=0
Sep 12 01:09:45.861288 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.861202    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"
Sep 12 01:09:45.861288 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.861247    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n" event={"ID":"e8a35c6d-185e-485a-bece-6da769a0f541","Type":"ContainerDied","Data":"fb4c9c01f7666c5cfe76ea6441792a091554e3271e868504d534ffeeb6ab3267"}
Sep 12 01:09:45.861391 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.861302    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n" event={"ID":"e8a35c6d-185e-485a-bece-6da769a0f541","Type":"ContainerDied","Data":"791d8cc176280568fcbd715eb7d11f14a52147d8bc9122d035ce601d8d57f51d"}
Sep 12 01:09:45.861391 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.861324    2567 scope.go:117] "RemoveContainer" containerID="fb4c9c01f7666c5cfe76ea6441792a091554e3271e868504d534ffeeb6ab3267"
Sep 12 01:09:45.871028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.871012    2567 scope.go:117] "RemoveContainer" containerID="fb4c9c01f7666c5cfe76ea6441792a091554e3271e868504d534ffeeb6ab3267"
Sep 12 01:09:45.871306 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:09:45.871287    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4c9c01f7666c5cfe76ea6441792a091554e3271e868504d534ffeeb6ab3267\": container with ID starting with fb4c9c01f7666c5cfe76ea6441792a091554e3271e868504d534ffeeb6ab3267 not found: ID does not exist" containerID="fb4c9c01f7666c5cfe76ea6441792a091554e3271e868504d534ffeeb6ab3267"
Sep 12 01:09:45.871379 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.871318    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4c9c01f7666c5cfe76ea6441792a091554e3271e868504d534ffeeb6ab3267"} err="failed to get container status \"fb4c9c01f7666c5cfe76ea6441792a091554e3271e868504d534ffeeb6ab3267\": rpc error: code = NotFound desc = could not find container \"fb4c9c01f7666c5cfe76ea6441792a091554e3271e868504d534ffeeb6ab3267\": container with ID starting with fb4c9c01f7666c5cfe76ea6441792a091554e3271e868504d534ffeeb6ab3267 not found: ID does not exist"
Sep 12 01:09:45.883911 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.883889    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"]
Sep 12 01:09:45.887774 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.887754    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-10ba6f-579c97f666-htj2n"]
Sep 12 01:09:45.975071 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:45.975003    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a35c6d-185e-485a-bece-6da769a0f541" path="/var/lib/kubelet/pods/e8a35c6d-185e-485a-bece-6da769a0f541/volumes"
Sep 12 01:09:46.540620 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:46.540573    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:47.403261 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:47.403218    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:48.091848 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.091815    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"]
Sep 12 01:09:48.092332 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.092092    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5" podUID="4c7e3cbe-53ef-4a7c-b217-de40db448969" containerName="maas-api" containerID="cri-o://b7750204fc387b7e79602576bb54e4cd4ba02b3036849ad0024894f0bffed52c" gracePeriod=30
Sep 12 01:09:48.121612 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.121583    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:48.201411 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.201348    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:48.299597 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.299568    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:48.342922 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.342869    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:09:48.398886 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.398860    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:48.406006 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.405938    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7e3cbe-53ef-4a7c-b217-de40db448969-metrics-certs\") pod \"4c7e3cbe-53ef-4a7c-b217-de40db448969\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") "
Sep 12 01:09:48.406133 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.406025    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c7e3cbe-53ef-4a7c-b217-de40db448969-openshift-service-ca\") pod \"4c7e3cbe-53ef-4a7c-b217-de40db448969\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") "
Sep 12 01:09:48.406133 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.406108    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9z9p\" (UniqueName: \"kubernetes.io/projected/4c7e3cbe-53ef-4a7c-b217-de40db448969-kube-api-access-d9z9p\") pod \"4c7e3cbe-53ef-4a7c-b217-de40db448969\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") "
Sep 12 01:09:48.406290 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.406138    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c7e3cbe-53ef-4a7c-b217-de40db448969-maas-api-tls\") pod \"4c7e3cbe-53ef-4a7c-b217-de40db448969\" (UID: \"4c7e3cbe-53ef-4a7c-b217-de40db448969\") "
Sep 12 01:09:48.406408 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.406382    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7e3cbe-53ef-4a7c-b217-de40db448969-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4c7e3cbe-53ef-4a7c-b217-de40db448969" (UID: "4c7e3cbe-53ef-4a7c-b217-de40db448969"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:48.406522 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.406507    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c7e3cbe-53ef-4a7c-b217-de40db448969-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:48.408347 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.408316    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7e3cbe-53ef-4a7c-b217-de40db448969-kube-api-access-d9z9p" (OuterVolumeSpecName: "kube-api-access-d9z9p") pod "4c7e3cbe-53ef-4a7c-b217-de40db448969" (UID: "4c7e3cbe-53ef-4a7c-b217-de40db448969"). InnerVolumeSpecName "kube-api-access-d9z9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:09:48.408445 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.408341    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c7e3cbe-53ef-4a7c-b217-de40db448969-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4c7e3cbe-53ef-4a7c-b217-de40db448969" (UID: "4c7e3cbe-53ef-4a7c-b217-de40db448969"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:09:48.408568 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.408543    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c7e3cbe-53ef-4a7c-b217-de40db448969-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4c7e3cbe-53ef-4a7c-b217-de40db448969" (UID: "4c7e3cbe-53ef-4a7c-b217-de40db448969"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:09:48.482276 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.482240    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:48.508082 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.508057    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9z9p\" (UniqueName: \"kubernetes.io/projected/4c7e3cbe-53ef-4a7c-b217-de40db448969-kube-api-access-d9z9p\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:48.508082 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.508081    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c7e3cbe-53ef-4a7c-b217-de40db448969-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:48.508256 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.508091    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7e3cbe-53ef-4a7c-b217-de40db448969-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:48.724086 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.724053    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:48.879244 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.879211    2567 generic.go:356] "Generic (PLEG): container finished" podID="4c7e3cbe-53ef-4a7c-b217-de40db448969" containerID="b7750204fc387b7e79602576bb54e4cd4ba02b3036849ad0024894f0bffed52c" exitCode=0
Sep 12 01:09:48.879401 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.879258    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5" event={"ID":"4c7e3cbe-53ef-4a7c-b217-de40db448969","Type":"ContainerDied","Data":"b7750204fc387b7e79602576bb54e4cd4ba02b3036849ad0024894f0bffed52c"}
Sep 12 01:09:48.879401 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.879275    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"
Sep 12 01:09:48.879401 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.879355    2567 scope.go:117] "RemoveContainer" containerID="b7750204fc387b7e79602576bb54e4cd4ba02b3036849ad0024894f0bffed52c"
Sep 12 01:09:48.879571 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.879345    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5" event={"ID":"4c7e3cbe-53ef-4a7c-b217-de40db448969","Type":"ContainerDied","Data":"82f6f375c36d2a8a9aea6de85d92402843e81aa0b058f14334f7c55695895cb2"}
Sep 12 01:09:48.889579 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.889560    2567 scope.go:117] "RemoveContainer" containerID="b7750204fc387b7e79602576bb54e4cd4ba02b3036849ad0024894f0bffed52c"
Sep 12 01:09:48.889858 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:09:48.889838    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7750204fc387b7e79602576bb54e4cd4ba02b3036849ad0024894f0bffed52c\": container with ID starting with b7750204fc387b7e79602576bb54e4cd4ba02b3036849ad0024894f0bffed52c not found: ID does not exist" containerID="b7750204fc387b7e79602576bb54e4cd4ba02b3036849ad0024894f0bffed52c"
Sep 12 01:09:48.889920 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.889871    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7750204fc387b7e79602576bb54e4cd4ba02b3036849ad0024894f0bffed52c"} err="failed to get container status \"b7750204fc387b7e79602576bb54e4cd4ba02b3036849ad0024894f0bffed52c\": rpc error: code = NotFound desc = could not find container \"b7750204fc387b7e79602576bb54e4cd4ba02b3036849ad0024894f0bffed52c\": container with ID starting with b7750204fc387b7e79602576bb54e4cd4ba02b3036849ad0024894f0bffed52c not found: ID does not exist"
Sep 12 01:09:48.903268 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.903243    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"]
Sep 12 01:09:48.905711 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:48.905684    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e62c2-fcc556bb5-hr5m5"]
Sep 12 01:09:49.191084 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.191036    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:49.407520 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.407480    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8" podUID="8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18" containerName="main" containerID="cri-o://4b40003a2ba9320a567ed5fd9087c6975fe5679fd0e1da6e0b5f1f77b40e125d" gracePeriod=45
Sep 12 01:09:49.416086 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.416056    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq" podUID="682f03ec-d30f-4129-89f9-9259b9c8bc77" containerName="main" containerID="cri-o://833e970fcbe394a8b7afda4b830ef91e1a267c55c46a2500ef022752c896339d" gracePeriod=45
Sep 12 01:09:49.685031 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.685004    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:09:49.690223 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.690198    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 01:09:49.820275 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820199    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-dshm\") pod \"682f03ec-d30f-4129-89f9-9259b9c8bc77\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") "
Sep 12 01:09:49.820275 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820236    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-model-cache\") pod \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") "
Sep 12 01:09:49.820275 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820263    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-home\") pod \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") "
Sep 12 01:09:49.820876 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820292    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-tmp-dir\") pod \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") "
Sep 12 01:09:49.820876 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820316    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-dshm\") pod \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") "
Sep 12 01:09:49.820876 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820358    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x7fs\" (UniqueName: \"kubernetes.io/projected/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-kube-api-access-7x7fs\") pod \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") "
Sep 12 01:09:49.820876 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820406    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-tmp-dir\") pod \"682f03ec-d30f-4129-89f9-9259b9c8bc77\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") "
Sep 12 01:09:49.820876 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820441    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-home\") pod \"682f03ec-d30f-4129-89f9-9259b9c8bc77\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") "
Sep 12 01:09:49.820876 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820487    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p28vh\" (UniqueName: \"kubernetes.io/projected/682f03ec-d30f-4129-89f9-9259b9c8bc77-kube-api-access-p28vh\") pod \"682f03ec-d30f-4129-89f9-9259b9c8bc77\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") "
Sep 12 01:09:49.820876 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820514    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-model-cache" (OuterVolumeSpecName: "model-cache") pod "8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18" (UID: "8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:09:49.820876 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820547    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-tls-certs\") pod \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\" (UID: \"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18\") "
Sep 12 01:09:49.820876 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820603    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18" (UID: "8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:09:49.820876 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820610    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-model-cache\") pod \"682f03ec-d30f-4129-89f9-9259b9c8bc77\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") "
Sep 12 01:09:49.820876 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820683    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "682f03ec-d30f-4129-89f9-9259b9c8bc77" (UID: "682f03ec-d30f-4129-89f9-9259b9c8bc77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:09:49.820876 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820705    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/682f03ec-d30f-4129-89f9-9259b9c8bc77-tls-certs\") pod \"682f03ec-d30f-4129-89f9-9259b9c8bc77\" (UID: \"682f03ec-d30f-4129-89f9-9259b9c8bc77\") "
Sep 12 01:09:49.820876 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820785    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-home" (OuterVolumeSpecName: "home") pod "8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18" (UID: "8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:09:49.820876 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.820789    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-model-cache" (OuterVolumeSpecName: "model-cache") pod "682f03ec-d30f-4129-89f9-9259b9c8bc77" (UID: "682f03ec-d30f-4129-89f9-9259b9c8bc77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:09:49.822007 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.821046    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-home" (OuterVolumeSpecName: "home") pod "682f03ec-d30f-4129-89f9-9259b9c8bc77" (UID: "682f03ec-d30f-4129-89f9-9259b9c8bc77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:09:49.822007 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.821181    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-tmp-dir\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:49.822007 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.821200    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-home\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:49.822007 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.821213    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-model-cache\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:49.822007 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.821225    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-model-cache\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:49.822007 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.821236    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-home\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:49.822007 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.821247    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-tmp-dir\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:49.823241 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.823205    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-dshm" (OuterVolumeSpecName: "dshm") pod "8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18" (UID: "8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:09:49.823357 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.823304    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-dshm" (OuterVolumeSpecName: "dshm") pod "682f03ec-d30f-4129-89f9-9259b9c8bc77" (UID: "682f03ec-d30f-4129-89f9-9259b9c8bc77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:09:49.823426 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.823378    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/682f03ec-d30f-4129-89f9-9259b9c8bc77-kube-api-access-p28vh" (OuterVolumeSpecName: "kube-api-access-p28vh") pod "682f03ec-d30f-4129-89f9-9259b9c8bc77" (UID: "682f03ec-d30f-4129-89f9-9259b9c8bc77"). InnerVolumeSpecName "kube-api-access-p28vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:09:49.823551 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.823521    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-kube-api-access-7x7fs" (OuterVolumeSpecName: "kube-api-access-7x7fs") pod "8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18" (UID: "8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18"). InnerVolumeSpecName "kube-api-access-7x7fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:09:49.823880 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.823855    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18" (UID: "8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:09:49.824260 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.824236    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/682f03ec-d30f-4129-89f9-9259b9c8bc77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "682f03ec-d30f-4129-89f9-9259b9c8bc77" (UID: "682f03ec-d30f-4129-89f9-9259b9c8bc77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:09:49.885064 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.885037    2567 generic.go:356] "Generic (PLEG): container finished" podID="8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18" containerID="4b40003a2ba9320a567ed5fd9087c6975fe5679fd0e1da6e0b5f1f77b40e125d" exitCode=0
Sep 12 01:09:49.885187 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.885085    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"
Sep 12 01:09:49.885187 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.885124    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8" event={"ID":"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18","Type":"ContainerDied","Data":"4b40003a2ba9320a567ed5fd9087c6975fe5679fd0e1da6e0b5f1f77b40e125d"}
Sep 12 01:09:49.885187 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.885169    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8" event={"ID":"8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18","Type":"ContainerDied","Data":"93ae6a3bbebe4a5fa261d28b5e054d4bb9859a8e1621f8654636d4c6052f0486"}
Sep 12 01:09:49.885321 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.885192    2567 scope.go:117] "RemoveContainer" containerID="4b40003a2ba9320a567ed5fd9087c6975fe5679fd0e1da6e0b5f1f77b40e125d"
Sep 12 01:09:49.887511 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.887491    2567 generic.go:356] "Generic (PLEG): container finished" podID="682f03ec-d30f-4129-89f9-9259b9c8bc77" containerID="833e970fcbe394a8b7afda4b830ef91e1a267c55c46a2500ef022752c896339d" exitCode=0
Sep 12 01:09:49.887620 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.887531    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"
Sep 12 01:09:49.887620 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.887544    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq" event={"ID":"682f03ec-d30f-4129-89f9-9259b9c8bc77","Type":"ContainerDied","Data":"833e970fcbe394a8b7afda4b830ef91e1a267c55c46a2500ef022752c896339d"}
Sep 12 01:09:49.887620 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.887589    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq" event={"ID":"682f03ec-d30f-4129-89f9-9259b9c8bc77","Type":"ContainerDied","Data":"5d67d6423616860c23703880f7171dc5126885a54b7ee064a1115c90a7d1fe8d"}
Sep 12 01:09:49.894786 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.894752    2567 scope.go:117] "RemoveContainer" containerID="4b40003a2ba9320a567ed5fd9087c6975fe5679fd0e1da6e0b5f1f77b40e125d"
Sep 12 01:09:49.895054 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:09:49.895036    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b40003a2ba9320a567ed5fd9087c6975fe5679fd0e1da6e0b5f1f77b40e125d\": container with ID starting with 4b40003a2ba9320a567ed5fd9087c6975fe5679fd0e1da6e0b5f1f77b40e125d not found: ID does not exist" containerID="4b40003a2ba9320a567ed5fd9087c6975fe5679fd0e1da6e0b5f1f77b40e125d"
Sep 12 01:09:49.895121 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.895063    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b40003a2ba9320a567ed5fd9087c6975fe5679fd0e1da6e0b5f1f77b40e125d"} err="failed to get container status \"4b40003a2ba9320a567ed5fd9087c6975fe5679fd0e1da6e0b5f1f77b40e125d\": rpc error: code = NotFound desc = could not find container \"4b40003a2ba9320a567ed5fd9087c6975fe5679fd0e1da6e0b5f1f77b40e125d\": container with ID starting with 4b40003a2ba9320a567ed5fd9087c6975fe5679fd0e1da6e0b5f1f77b40e125d not found: ID does not exist"
Sep 12 01:09:49.895121 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.895079    2567 scope.go:117] "RemoveContainer" containerID="833e970fcbe394a8b7afda4b830ef91e1a267c55c46a2500ef022752c896339d"
Sep 12 01:09:49.902756 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.902739    2567 scope.go:117] "RemoveContainer" containerID="833e970fcbe394a8b7afda4b830ef91e1a267c55c46a2500ef022752c896339d"
Sep 12 01:09:49.902999 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:09:49.902974    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833e970fcbe394a8b7afda4b830ef91e1a267c55c46a2500ef022752c896339d\": container with ID starting with 833e970fcbe394a8b7afda4b830ef91e1a267c55c46a2500ef022752c896339d not found: ID does not exist" containerID="833e970fcbe394a8b7afda4b830ef91e1a267c55c46a2500ef022752c896339d"
Sep 12 01:09:49.903093 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.903005    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833e970fcbe394a8b7afda4b830ef91e1a267c55c46a2500ef022752c896339d"} err="failed to get container status \"833e970fcbe394a8b7afda4b830ef91e1a267c55c46a2500ef022752c896339d\": rpc error: code = NotFound desc = could not find container \"833e970fcbe394a8b7afda4b830ef91e1a267c55c46a2500ef022752c896339d\": container with ID starting with 833e970fcbe394a8b7afda4b830ef91e1a267c55c46a2500ef022752c896339d not found: ID does not exist"
Sep 12 01:09:49.907400 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.907381    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"]
Sep 12 01:09:49.912018 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.911998    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-10ba6f/embedding-w0-10ba6f-kserve-54786fb55b-cpcb8"]
Sep 12 01:09:49.922480 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.922461    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/682f03ec-d30f-4129-89f9-9259b9c8bc77-dshm\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:49.922480 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.922482    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-dshm\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:49.922605 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.922491    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x7fs\" (UniqueName: \"kubernetes.io/projected/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-kube-api-access-7x7fs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:49.922605 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.922501    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p28vh\" (UniqueName: \"kubernetes.io/projected/682f03ec-d30f-4129-89f9-9259b9c8bc77-kube-api-access-p28vh\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:49.922605 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.922510    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18-tls-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:49.922605 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.922517    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/682f03ec-d30f-4129-89f9-9259b9c8bc77-tls-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:49.923097 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.923079    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"]
Sep 12 01:09:49.928602 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.928583    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-10ba6f/premium-w0-10ba6f-kserve-59f9bf6ff6-vkqjq"]
Sep 12 01:09:49.973763 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.973740    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c7e3cbe-53ef-4a7c-b217-de40db448969" path="/var/lib/kubelet/pods/4c7e3cbe-53ef-4a7c-b217-de40db448969/volumes"
Sep 12 01:09:49.974154 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.974142    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="682f03ec-d30f-4129-89f9-9259b9c8bc77" path="/var/lib/kubelet/pods/682f03ec-d30f-4129-89f9-9259b9c8bc77/volumes"
Sep 12 01:09:49.974482 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:49.974471    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18" path="/var/lib/kubelet/pods/8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18/volumes"
Sep 12 01:09:50.088467 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:50.088404    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:50.891195 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:50.891168    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:51.590604 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:51.590573    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:51.841970 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:51.841874    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:51.950470 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:51.950440    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:52.096591 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:52.096512    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:52.412696 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:52.412671    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:54.746822 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:54.746787    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:54.815717 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:54.815684    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:55.012657 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:55.012558    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:55.087601 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:55.087562    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:55.558897 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:55.558868    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:56.336508 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:56.336478    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:57.002830 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:57.002800    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:57.807914 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:57.807882    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:58.597995 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:58.597968    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:09:59.369240 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:09:59.369211    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:00.076635 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:00.076601    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"]
Sep 12 01:10:00.083491 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:00.083459    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"]
Sep 12 01:10:00.096335 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:00.096305    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"]
Sep 12 01:10:00.153432 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:00.153396    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:05.727685 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:05.727642    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"]
Sep 12 01:10:05.728163 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:05.727931    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf" podUID="e6912507-9a44-4218-97ff-0d226d246990" containerName="maas-api" containerID="cri-o://02129ba15633fb586b54072dd7a824cabaa476ee913756cdad49a9170c1b151a" gracePeriod=30
Sep 12 01:10:05.756309 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:05.756285    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:05.824076 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:05.824043    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:05.925076 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:05.925042    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:05.970496 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:05.970469    2567 generic.go:356] "Generic (PLEG): container finished" podID="e6912507-9a44-4218-97ff-0d226d246990" containerID="02129ba15633fb586b54072dd7a824cabaa476ee913756cdad49a9170c1b151a" exitCode=0
Sep 12 01:10:05.974777 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:05.974748    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf" event={"ID":"e6912507-9a44-4218-97ff-0d226d246990","Type":"ContainerDied","Data":"02129ba15633fb586b54072dd7a824cabaa476ee913756cdad49a9170c1b151a"}
Sep 12 01:10:05.974884 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:05.974794    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf" event={"ID":"e6912507-9a44-4218-97ff-0d226d246990","Type":"ContainerDied","Data":"e4ac1153cbc53c058d1417cc27c10c5bd6890d8a7c0faa291594478699ada569"}
Sep 12 01:10:05.974884 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:05.974811    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4ac1153cbc53c058d1417cc27c10c5bd6890d8a7c0faa291594478699ada569"
Sep 12 01:10:05.984764 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:05.984711    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:10:06.008174 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.008140    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:06.075662 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.075633    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:06.077796 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.077778    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-maas-api-tls\") pod \"e6912507-9a44-4218-97ff-0d226d246990\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") "
Sep 12 01:10:06.077895 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.077808    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-metrics-certs\") pod \"e6912507-9a44-4218-97ff-0d226d246990\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") "
Sep 12 01:10:06.077895 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.077843    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t9d8\" (UniqueName: \"kubernetes.io/projected/e6912507-9a44-4218-97ff-0d226d246990-kube-api-access-4t9d8\") pod \"e6912507-9a44-4218-97ff-0d226d246990\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") "
Sep 12 01:10:06.079996 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.079963    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e6912507-9a44-4218-97ff-0d226d246990" (UID: "e6912507-9a44-4218-97ff-0d226d246990"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:10:06.080090 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.080045    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6912507-9a44-4218-97ff-0d226d246990-kube-api-access-4t9d8" (OuterVolumeSpecName: "kube-api-access-4t9d8") pod "e6912507-9a44-4218-97ff-0d226d246990" (UID: "e6912507-9a44-4218-97ff-0d226d246990"). InnerVolumeSpecName "kube-api-access-4t9d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:10:06.080158 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.080137    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e6912507-9a44-4218-97ff-0d226d246990" (UID: "e6912507-9a44-4218-97ff-0d226d246990"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:10:06.178985 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.178926    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6912507-9a44-4218-97ff-0d226d246990-openshift-service-ca\") pod \"e6912507-9a44-4218-97ff-0d226d246990\" (UID: \"e6912507-9a44-4218-97ff-0d226d246990\") "
Sep 12 01:10:06.179223 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.179210    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:06.179272 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.179225    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6912507-9a44-4218-97ff-0d226d246990-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:06.179272 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.179227    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6912507-9a44-4218-97ff-0d226d246990-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e6912507-9a44-4218-97ff-0d226d246990" (UID: "e6912507-9a44-4218-97ff-0d226d246990"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:10:06.179272 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.179235    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9d8\" (UniqueName: \"kubernetes.io/projected/e6912507-9a44-4218-97ff-0d226d246990-kube-api-access-4t9d8\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:06.279825 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.279741    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6912507-9a44-4218-97ff-0d226d246990-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:06.498941 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.498913    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:06.599261 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.599168    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:06.977974 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:06.976648    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"
Sep 12 01:10:07.007435 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:07.007402    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"]
Sep 12 01:10:07.011816 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:07.011792    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-18b9d7-5d7b6df679-2rvtf"]
Sep 12 01:10:07.478066 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:07.478034    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:07.973852 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:07.973825    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6912507-9a44-4218-97ff-0d226d246990" path="/var/lib/kubelet/pods/e6912507-9a44-4218-97ff-0d226d246990/volumes"
Sep 12 01:10:08.053497 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.053467    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"]
Sep 12 01:10:08.053941 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.053759    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh" podUID="778eaff8-6d09-41d7-812f-7ab5a2e4636b" containerName="maas-api" containerID="cri-o://3a71b6436514bb569fdab24afc141165b459b69936ae9d84f34af57b8b70dbba" gracePeriod=30
Sep 12 01:10:08.077600 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.077560    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:08.154774 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.154732    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:08.262042 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.261935    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:08.306509 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.306486    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 01:10:08.344610 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.344579    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:08.396683 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.396644    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-maas-api-tls\") pod \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") "
Sep 12 01:10:08.396847 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.396687    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-metrics-certs\") pod \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") "
Sep 12 01:10:08.396847 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.396765    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/778eaff8-6d09-41d7-812f-7ab5a2e4636b-openshift-service-ca\") pod \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") "
Sep 12 01:10:08.397057 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.396888    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnkn6\" (UniqueName: \"kubernetes.io/projected/778eaff8-6d09-41d7-812f-7ab5a2e4636b-kube-api-access-fnkn6\") pod \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\" (UID: \"778eaff8-6d09-41d7-812f-7ab5a2e4636b\") "
Sep 12 01:10:08.397209 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.397179    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778eaff8-6d09-41d7-812f-7ab5a2e4636b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "778eaff8-6d09-41d7-812f-7ab5a2e4636b" (UID: "778eaff8-6d09-41d7-812f-7ab5a2e4636b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:10:08.397325 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.397286    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/778eaff8-6d09-41d7-812f-7ab5a2e4636b-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:08.399098 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.399073    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "778eaff8-6d09-41d7-812f-7ab5a2e4636b" (UID: "778eaff8-6d09-41d7-812f-7ab5a2e4636b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:10:08.399199 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.399152    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778eaff8-6d09-41d7-812f-7ab5a2e4636b-kube-api-access-fnkn6" (OuterVolumeSpecName: "kube-api-access-fnkn6") pod "778eaff8-6d09-41d7-812f-7ab5a2e4636b" (UID: "778eaff8-6d09-41d7-812f-7ab5a2e4636b"). InnerVolumeSpecName "kube-api-access-fnkn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:10:08.399199 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.399162    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "778eaff8-6d09-41d7-812f-7ab5a2e4636b" (UID: "778eaff8-6d09-41d7-812f-7ab5a2e4636b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:10:08.413932 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.413911    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:08.497890 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.497856    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnkn6\" (UniqueName: \"kubernetes.io/projected/778eaff8-6d09-41d7-812f-7ab5a2e4636b-kube-api-access-fnkn6\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:08.497890 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.497884    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:08.497890 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.497894    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/778eaff8-6d09-41d7-812f-7ab5a2e4636b-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:08.560759 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.560682    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:08.704281 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.704251    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:08.988528 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.988492    2567 generic.go:356] "Generic (PLEG): container finished" podID="778eaff8-6d09-41d7-812f-7ab5a2e4636b" containerID="3a71b6436514bb569fdab24afc141165b459b69936ae9d84f34af57b8b70dbba" exitCode=0
Sep 12 01:10:08.988691 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.988535    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"
Sep 12 01:10:08.988691 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.988557    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh" event={"ID":"778eaff8-6d09-41d7-812f-7ab5a2e4636b","Type":"ContainerDied","Data":"3a71b6436514bb569fdab24afc141165b459b69936ae9d84f34af57b8b70dbba"}
Sep 12 01:10:08.988691 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.988602    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh" event={"ID":"778eaff8-6d09-41d7-812f-7ab5a2e4636b","Type":"ContainerDied","Data":"dec5067e7be3fe13aefd573032b1451ba15b54ea62f2d81bfb22755d7e51be0e"}
Sep 12 01:10:08.988691 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.988628    2567 scope.go:117] "RemoveContainer" containerID="3a71b6436514bb569fdab24afc141165b459b69936ae9d84f34af57b8b70dbba"
Sep 12 01:10:08.998089 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.998072    2567 scope.go:117] "RemoveContainer" containerID="3a71b6436514bb569fdab24afc141165b459b69936ae9d84f34af57b8b70dbba"
Sep 12 01:10:08.998357 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:10:08.998337    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a71b6436514bb569fdab24afc141165b459b69936ae9d84f34af57b8b70dbba\": container with ID starting with 3a71b6436514bb569fdab24afc141165b459b69936ae9d84f34af57b8b70dbba not found: ID does not exist" containerID="3a71b6436514bb569fdab24afc141165b459b69936ae9d84f34af57b8b70dbba"
Sep 12 01:10:08.998406 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:08.998367    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a71b6436514bb569fdab24afc141165b459b69936ae9d84f34af57b8b70dbba"} err="failed to get container status \"3a71b6436514bb569fdab24afc141165b459b69936ae9d84f34af57b8b70dbba\": rpc error: code = NotFound desc = could not find container \"3a71b6436514bb569fdab24afc141165b459b69936ae9d84f34af57b8b70dbba\": container with ID starting with 3a71b6436514bb569fdab24afc141165b459b69936ae9d84f34af57b8b70dbba not found: ID does not exist"
Sep 12 01:10:09.010180 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:09.010156    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"]
Sep 12 01:10:09.013770 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:09.013751    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f19fd1-7fdfb5b657-gzjrh"]
Sep 12 01:10:09.380456 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:09.380378    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:09.973777 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:09.973743    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="778eaff8-6d09-41d7-812f-7ab5a2e4636b" path="/var/lib/kubelet/pods/778eaff8-6d09-41d7-812f-7ab5a2e4636b/volumes"
Sep 12 01:10:10.306919 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:10.306832    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:10.983123 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:10.983092    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:11.685975 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:11.685890    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:11.784632 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:11.784592    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:12.237346 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:12.237313    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:12.788083 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:12.788047    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:12.851436 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:12.851399    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:13.041865 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:13.041767    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:14.947892 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:14.947846    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:15.010673 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.010639    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:15.096709 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.096672    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5" podUID="6764424e-95bf-41a6-ae53-e6cb4b696731" containerName="main" containerID="cri-o://859df7460f201d492fafe7670bfa57f9a68c9d758ce883af9c43322266def1e1" gracePeriod=45
Sep 12 01:10:15.103296 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.103269    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78" podUID="a20ac225-20b4-4ec7-8a8b-c2c47a3bb747" containerName="main" containerID="cri-o://444764f0fc6f03fe6b4cf72a7027724cfe5f73645c5b9410a8e38a25ff9725a9" gracePeriod=45
Sep 12 01:10:15.115581 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.115555    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f" podUID="91095c2f-1f53-4b78-b74e-fd8c1a9e962e" containerName="main" containerID="cri-o://a5484409d6e534867345fdb32bbd0ff981d65b81586e0bb3b36d8b6df2a1a9ef" gracePeriod=45
Sep 12 01:10:15.203903 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.203839    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:15.365528 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.365503    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:10:15.417873 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.417841    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 01:10:15.421360 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.421339    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 01:10:15.458475 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458343    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-dshm\") pod \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") "
Sep 12 01:10:15.458475 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458386    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-model-cache\") pod \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") "
Sep 12 01:10:15.458475 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458431    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-tmp-dir\") pod \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") "
Sep 12 01:10:15.458475 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458459    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-dshm\") pod \"6764424e-95bf-41a6-ae53-e6cb4b696731\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") "
Sep 12 01:10:15.458778 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458493    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd86k\" (UniqueName: \"kubernetes.io/projected/6764424e-95bf-41a6-ae53-e6cb4b696731-kube-api-access-rd86k\") pod \"6764424e-95bf-41a6-ae53-e6cb4b696731\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") "
Sep 12 01:10:15.458778 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458527    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-tmp-dir\") pod \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") "
Sep 12 01:10:15.458778 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458561    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql5s4\" (UniqueName: \"kubernetes.io/projected/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-kube-api-access-ql5s4\") pod \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") "
Sep 12 01:10:15.458778 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458598    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-home\") pod \"6764424e-95bf-41a6-ae53-e6cb4b696731\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") "
Sep 12 01:10:15.458778 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458636    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-model-cache\") pod \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") "
Sep 12 01:10:15.458778 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458658    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-tmp-dir\") pod \"6764424e-95bf-41a6-ae53-e6cb4b696731\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") "
Sep 12 01:10:15.458778 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458684    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6764424e-95bf-41a6-ae53-e6cb4b696731-tls-certs\") pod \"6764424e-95bf-41a6-ae53-e6cb4b696731\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") "
Sep 12 01:10:15.458778 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458715    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-tls-certs\") pod \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") "
Sep 12 01:10:15.458778 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458741    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-dshm\") pod \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") "
Sep 12 01:10:15.458778 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458769    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-model-cache\") pod \"6764424e-95bf-41a6-ae53-e6cb4b696731\" (UID: \"6764424e-95bf-41a6-ae53-e6cb4b696731\") "
Sep 12 01:10:15.459310 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458806    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-home\") pod \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") "
Sep 12 01:10:15.459310 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458850    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt652\" (UniqueName: \"kubernetes.io/projected/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-kube-api-access-nt652\") pod \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") "
Sep 12 01:10:15.459310 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458876    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-home\") pod \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\" (UID: \"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747\") "
Sep 12 01:10:15.459310 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.458995    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-tls-certs\") pod \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\" (UID: \"91095c2f-1f53-4b78-b74e-fd8c1a9e962e\") "
Sep 12 01:10:15.459525 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.459306    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91095c2f-1f53-4b78-b74e-fd8c1a9e962e" (UID: "91095c2f-1f53-4b78-b74e-fd8c1a9e962e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:15.459525 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.459373    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6764424e-95bf-41a6-ae53-e6cb4b696731" (UID: "6764424e-95bf-41a6-ae53-e6cb4b696731"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:15.459525 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.459425    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-tmp-dir\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.459682 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.459586    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-model-cache" (OuterVolumeSpecName: "model-cache") pod "91095c2f-1f53-4b78-b74e-fd8c1a9e962e" (UID: "91095c2f-1f53-4b78-b74e-fd8c1a9e962e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:15.460526 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.460497    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-model-cache" (OuterVolumeSpecName: "model-cache") pod "a20ac225-20b4-4ec7-8a8b-c2c47a3bb747" (UID: "a20ac225-20b4-4ec7-8a8b-c2c47a3bb747"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:15.460799 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.460774    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-home" (OuterVolumeSpecName: "home") pod "6764424e-95bf-41a6-ae53-e6cb4b696731" (UID: "6764424e-95bf-41a6-ae53-e6cb4b696731"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:15.461099 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.461068    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-model-cache" (OuterVolumeSpecName: "model-cache") pod "6764424e-95bf-41a6-ae53-e6cb4b696731" (UID: "6764424e-95bf-41a6-ae53-e6cb4b696731"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:15.461377 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.461353    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a20ac225-20b4-4ec7-8a8b-c2c47a3bb747" (UID: "a20ac225-20b4-4ec7-8a8b-c2c47a3bb747"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:15.461854 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.461817    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-dshm" (OuterVolumeSpecName: "dshm") pod "91095c2f-1f53-4b78-b74e-fd8c1a9e962e" (UID: "91095c2f-1f53-4b78-b74e-fd8c1a9e962e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:15.462048 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.462027    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-home" (OuterVolumeSpecName: "home") pod "a20ac225-20b4-4ec7-8a8b-c2c47a3bb747" (UID: "a20ac225-20b4-4ec7-8a8b-c2c47a3bb747"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:15.462138 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.462076    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-home" (OuterVolumeSpecName: "home") pod "91095c2f-1f53-4b78-b74e-fd8c1a9e962e" (UID: "91095c2f-1f53-4b78-b74e-fd8c1a9e962e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:15.462599 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.462579    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91095c2f-1f53-4b78-b74e-fd8c1a9e962e" (UID: "91095c2f-1f53-4b78-b74e-fd8c1a9e962e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:10:15.462687 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.462619    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-kube-api-access-ql5s4" (OuterVolumeSpecName: "kube-api-access-ql5s4") pod "a20ac225-20b4-4ec7-8a8b-c2c47a3bb747" (UID: "a20ac225-20b4-4ec7-8a8b-c2c47a3bb747"). InnerVolumeSpecName "kube-api-access-ql5s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:10:15.464067 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.464036    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6764424e-95bf-41a6-ae53-e6cb4b696731-kube-api-access-rd86k" (OuterVolumeSpecName: "kube-api-access-rd86k") pod "6764424e-95bf-41a6-ae53-e6cb4b696731" (UID: "6764424e-95bf-41a6-ae53-e6cb4b696731"). InnerVolumeSpecName "kube-api-access-rd86k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:10:15.464170 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.464095    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-dshm" (OuterVolumeSpecName: "dshm") pod "6764424e-95bf-41a6-ae53-e6cb4b696731" (UID: "6764424e-95bf-41a6-ae53-e6cb4b696731"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:15.464271 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.464249    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6764424e-95bf-41a6-ae53-e6cb4b696731-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6764424e-95bf-41a6-ae53-e6cb4b696731" (UID: "6764424e-95bf-41a6-ae53-e6cb4b696731"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:10:15.464429 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.464412    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-kube-api-access-nt652" (OuterVolumeSpecName: "kube-api-access-nt652") pod "91095c2f-1f53-4b78-b74e-fd8c1a9e962e" (UID: "91095c2f-1f53-4b78-b74e-fd8c1a9e962e"). InnerVolumeSpecName "kube-api-access-nt652". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:10:15.464509 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.464441    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a20ac225-20b4-4ec7-8a8b-c2c47a3bb747" (UID: "a20ac225-20b4-4ec7-8a8b-c2c47a3bb747"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:10:15.465525 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.465507    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-dshm" (OuterVolumeSpecName: "dshm") pod "a20ac225-20b4-4ec7-8a8b-c2c47a3bb747" (UID: "a20ac225-20b4-4ec7-8a8b-c2c47a3bb747"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:15.560740 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560697    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-home\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.560740 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560735    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt652\" (UniqueName: \"kubernetes.io/projected/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-kube-api-access-nt652\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.560740 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560747    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-home\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.560740 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560755    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-tls-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.561028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560763    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-dshm\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.561028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560770    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91095c2f-1f53-4b78-b74e-fd8c1a9e962e-model-cache\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.561028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560777    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-dshm\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.561028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560785    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd86k\" (UniqueName: \"kubernetes.io/projected/6764424e-95bf-41a6-ae53-e6cb4b696731-kube-api-access-rd86k\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.561028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560792    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-tmp-dir\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.561028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560801    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql5s4\" (UniqueName: \"kubernetes.io/projected/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-kube-api-access-ql5s4\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.561028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560809    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-home\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.561028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560817    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-model-cache\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.561028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560824    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-tmp-dir\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.561028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560832    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6764424e-95bf-41a6-ae53-e6cb4b696731-tls-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.561028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560839    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-tls-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.561028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560845    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747-dshm\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:15.561028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:15.560852    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6764424e-95bf-41a6-ae53-e6cb4b696731-model-cache\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:16.027039 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.027012    2567 generic.go:356] "Generic (PLEG): container finished" podID="a20ac225-20b4-4ec7-8a8b-c2c47a3bb747" containerID="444764f0fc6f03fe6b4cf72a7027724cfe5f73645c5b9410a8e38a25ff9725a9" exitCode=0
Sep 12 01:10:16.027373 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.027052    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"
Sep 12 01:10:16.027373 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.027095    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78" event={"ID":"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747","Type":"ContainerDied","Data":"444764f0fc6f03fe6b4cf72a7027724cfe5f73645c5b9410a8e38a25ff9725a9"}
Sep 12 01:10:16.027373 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.027133    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78" event={"ID":"a20ac225-20b4-4ec7-8a8b-c2c47a3bb747","Type":"ContainerDied","Data":"d93967aaacd4f4a6301ec40077783c4bfdb3f0c804b8fe978d07a0f8f739661d"}
Sep 12 01:10:16.027373 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.027151    2567 scope.go:117] "RemoveContainer" containerID="444764f0fc6f03fe6b4cf72a7027724cfe5f73645c5b9410a8e38a25ff9725a9"
Sep 12 01:10:16.028505 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.028485    2567 generic.go:356] "Generic (PLEG): container finished" podID="91095c2f-1f53-4b78-b74e-fd8c1a9e962e" containerID="a5484409d6e534867345fdb32bbd0ff981d65b81586e0bb3b36d8b6df2a1a9ef" exitCode=0
Sep 12 01:10:16.028654 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.028518    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"
Sep 12 01:10:16.028654 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.028574    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f" event={"ID":"91095c2f-1f53-4b78-b74e-fd8c1a9e962e","Type":"ContainerDied","Data":"a5484409d6e534867345fdb32bbd0ff981d65b81586e0bb3b36d8b6df2a1a9ef"}
Sep 12 01:10:16.028654 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.028610    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f" event={"ID":"91095c2f-1f53-4b78-b74e-fd8c1a9e962e","Type":"ContainerDied","Data":"97c35bbe3256bb6b826a8b7d5e46c0ccc6cd5e7070a5b4c87727c8928e52cf17"}
Sep 12 01:10:16.030585 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.030563    2567 generic.go:356] "Generic (PLEG): container finished" podID="6764424e-95bf-41a6-ae53-e6cb4b696731" containerID="859df7460f201d492fafe7670bfa57f9a68c9d758ce883af9c43322266def1e1" exitCode=0
Sep 12 01:10:16.030698 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.030610    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5" event={"ID":"6764424e-95bf-41a6-ae53-e6cb4b696731","Type":"ContainerDied","Data":"859df7460f201d492fafe7670bfa57f9a68c9d758ce883af9c43322266def1e1"}
Sep 12 01:10:16.030698 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.030639    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5" event={"ID":"6764424e-95bf-41a6-ae53-e6cb4b696731","Type":"ContainerDied","Data":"881693f7f0eaaf8cac57e9af0b91a502af2f97e0830fc85015e0b7de4f7c1456"}
Sep 12 01:10:16.030860 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.030837    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"
Sep 12 01:10:16.037102 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.037086    2567 scope.go:117] "RemoveContainer" containerID="444764f0fc6f03fe6b4cf72a7027724cfe5f73645c5b9410a8e38a25ff9725a9"
Sep 12 01:10:16.037397 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:10:16.037377    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"444764f0fc6f03fe6b4cf72a7027724cfe5f73645c5b9410a8e38a25ff9725a9\": container with ID starting with 444764f0fc6f03fe6b4cf72a7027724cfe5f73645c5b9410a8e38a25ff9725a9 not found: ID does not exist" containerID="444764f0fc6f03fe6b4cf72a7027724cfe5f73645c5b9410a8e38a25ff9725a9"
Sep 12 01:10:16.037475 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.037409    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444764f0fc6f03fe6b4cf72a7027724cfe5f73645c5b9410a8e38a25ff9725a9"} err="failed to get container status \"444764f0fc6f03fe6b4cf72a7027724cfe5f73645c5b9410a8e38a25ff9725a9\": rpc error: code = NotFound desc = could not find container \"444764f0fc6f03fe6b4cf72a7027724cfe5f73645c5b9410a8e38a25ff9725a9\": container with ID starting with 444764f0fc6f03fe6b4cf72a7027724cfe5f73645c5b9410a8e38a25ff9725a9 not found: ID does not exist"
Sep 12 01:10:16.037475 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.037431    2567 scope.go:117] "RemoveContainer" containerID="a5484409d6e534867345fdb32bbd0ff981d65b81586e0bb3b36d8b6df2a1a9ef"
Sep 12 01:10:16.045078 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.045028    2567 scope.go:117] "RemoveContainer" containerID="a5484409d6e534867345fdb32bbd0ff981d65b81586e0bb3b36d8b6df2a1a9ef"
Sep 12 01:10:16.045362 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:10:16.045341    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5484409d6e534867345fdb32bbd0ff981d65b81586e0bb3b36d8b6df2a1a9ef\": container with ID starting with a5484409d6e534867345fdb32bbd0ff981d65b81586e0bb3b36d8b6df2a1a9ef not found: ID does not exist" containerID="a5484409d6e534867345fdb32bbd0ff981d65b81586e0bb3b36d8b6df2a1a9ef"
Sep 12 01:10:16.045439 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.045368    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5484409d6e534867345fdb32bbd0ff981d65b81586e0bb3b36d8b6df2a1a9ef"} err="failed to get container status \"a5484409d6e534867345fdb32bbd0ff981d65b81586e0bb3b36d8b6df2a1a9ef\": rpc error: code = NotFound desc = could not find container \"a5484409d6e534867345fdb32bbd0ff981d65b81586e0bb3b36d8b6df2a1a9ef\": container with ID starting with a5484409d6e534867345fdb32bbd0ff981d65b81586e0bb3b36d8b6df2a1a9ef not found: ID does not exist"
Sep 12 01:10:16.045439 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.045382    2567 scope.go:117] "RemoveContainer" containerID="859df7460f201d492fafe7670bfa57f9a68c9d758ce883af9c43322266def1e1"
Sep 12 01:10:16.053829 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.053811    2567 scope.go:117] "RemoveContainer" containerID="859df7460f201d492fafe7670bfa57f9a68c9d758ce883af9c43322266def1e1"
Sep 12 01:10:16.054055 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.054036    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"]
Sep 12 01:10:16.054148 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:10:16.054128    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859df7460f201d492fafe7670bfa57f9a68c9d758ce883af9c43322266def1e1\": container with ID starting with 859df7460f201d492fafe7670bfa57f9a68c9d758ce883af9c43322266def1e1 not found: ID does not exist" containerID="859df7460f201d492fafe7670bfa57f9a68c9d758ce883af9c43322266def1e1"
Sep 12 01:10:16.054197 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.054153    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859df7460f201d492fafe7670bfa57f9a68c9d758ce883af9c43322266def1e1"} err="failed to get container status \"859df7460f201d492fafe7670bfa57f9a68c9d758ce883af9c43322266def1e1\": rpc error: code = NotFound desc = could not find container \"859df7460f201d492fafe7670bfa57f9a68c9d758ce883af9c43322266def1e1\": container with ID starting with 859df7460f201d492fafe7670bfa57f9a68c9d758ce883af9c43322266def1e1 not found: ID does not exist"
Sep 12 01:10:16.058131 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.058108    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/premium-w4-f19fd1-kserve-589c4df7bc-nsr78"]
Sep 12 01:10:16.069214 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.069180    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"]
Sep 12 01:10:16.070817 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.070798    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/distinct-w4-f19fd1-kserve-765cd874f8-wv6v5"]
Sep 12 01:10:16.083189 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.083167    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"]
Sep 12 01:10:16.085371 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.085352    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f19fd1/simulator-w4-f19fd1-kserve-69694947cd-t5m7f"]
Sep 12 01:10:16.090890 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:16.090872    2567 scope.go:117] "RemoveContainer" containerID="02129ba15633fb586b54072dd7a824cabaa476ee913756cdad49a9170c1b151a"
Sep 12 01:10:17.803627 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:17.803582    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:17.974306 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:17.974275    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6764424e-95bf-41a6-ae53-e6cb4b696731" path="/var/lib/kubelet/pods/6764424e-95bf-41a6-ae53-e6cb4b696731/volumes"
Sep 12 01:10:17.974662 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:17.974650    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91095c2f-1f53-4b78-b74e-fd8c1a9e962e" path="/var/lib/kubelet/pods/91095c2f-1f53-4b78-b74e-fd8c1a9e962e/volumes"
Sep 12 01:10:17.975000 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:17.974988    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a20ac225-20b4-4ec7-8a8b-c2c47a3bb747" path="/var/lib/kubelet/pods/a20ac225-20b4-4ec7-8a8b-c2c47a3bb747/volumes"
Sep 12 01:10:23.731586 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:23.731551    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:42.977943 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.977909    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8023-64a496"]
Sep 12 01:10:42.978322 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978273    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6912507-9a44-4218-97ff-0d226d246990" containerName="maas-api"
Sep 12 01:10:42.978322 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978310    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6912507-9a44-4218-97ff-0d226d246990" containerName="maas-api"
Sep 12 01:10:42.978398 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978324    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c7e3cbe-53ef-4a7c-b217-de40db448969" containerName="maas-api"
Sep 12 01:10:42.978398 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978329    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c7e3cbe-53ef-4a7c-b217-de40db448969" containerName="maas-api"
Sep 12 01:10:42.978398 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978340    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="682f03ec-d30f-4129-89f9-9259b9c8bc77" containerName="main"
Sep 12 01:10:42.978398 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978345    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="682f03ec-d30f-4129-89f9-9259b9c8bc77" containerName="main"
Sep 12 01:10:42.978398 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978353    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18" containerName="main"
Sep 12 01:10:42.978398 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978359    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18" containerName="main"
Sep 12 01:10:42.978398 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978372    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778eaff8-6d09-41d7-812f-7ab5a2e4636b" containerName="maas-api"
Sep 12 01:10:42.978398 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978376    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="778eaff8-6d09-41d7-812f-7ab5a2e4636b" containerName="maas-api"
Sep 12 01:10:42.978398 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978383    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a20ac225-20b4-4ec7-8a8b-c2c47a3bb747" containerName="main"
Sep 12 01:10:42.978398 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978389    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a20ac225-20b4-4ec7-8a8b-c2c47a3bb747" containerName="main"
Sep 12 01:10:42.978398 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978395    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2f35eb0-2d56-4141-b204-8cf8f0d35dbd" containerName="maas-api"
Sep 12 01:10:42.978398 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978400    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2f35eb0-2d56-4141-b204-8cf8f0d35dbd" containerName="maas-api"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978408    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6764424e-95bf-41a6-ae53-e6cb4b696731" containerName="main"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978412    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6764424e-95bf-41a6-ae53-e6cb4b696731" containerName="main"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978418    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a35c6d-185e-485a-bece-6da769a0f541" containerName="maas-api"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978423    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a35c6d-185e-485a-bece-6da769a0f541" containerName="maas-api"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978434    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a85529a6-3f04-498a-a8a7-da73662c5f67" containerName="maas-api"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978438    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a85529a6-3f04-498a-a8a7-da73662c5f67" containerName="maas-api"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978443    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91095c2f-1f53-4b78-b74e-fd8c1a9e962e" containerName="main"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978448    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="91095c2f-1f53-4b78-b74e-fd8c1a9e962e" containerName="main"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978515    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="91095c2f-1f53-4b78-b74e-fd8c1a9e962e" containerName="main"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978521    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6912507-9a44-4218-97ff-0d226d246990" containerName="maas-api"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978527    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6764424e-95bf-41a6-ae53-e6cb4b696731" containerName="main"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978531    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="682f03ec-d30f-4129-89f9-9259b9c8bc77" containerName="main"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978538    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a85529a6-3f04-498a-a8a7-da73662c5f67" containerName="maas-api"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978543    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="778eaff8-6d09-41d7-812f-7ab5a2e4636b" containerName="maas-api"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978549    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a20ac225-20b4-4ec7-8a8b-c2c47a3bb747" containerName="main"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978555    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2f35eb0-2d56-4141-b204-8cf8f0d35dbd" containerName="maas-api"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978560    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c7e3cbe-53ef-4a7c-b217-de40db448969" containerName="maas-api"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978565    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a35c6d-185e-485a-bece-6da769a0f541" containerName="maas-api"
Sep 12 01:10:42.978762 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.978570    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dd44c11-5a0e-4a4a-8a9c-9c45ce4b7a18" containerName="main"
Sep 12 01:10:42.983537 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.983512    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8023-64a496"
Sep 12 01:10:42.985845 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.985817    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8023-64a496"]
Sep 12 01:10:42.986156 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:42.986137    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n87bf\""
Sep 12 01:10:43.119364 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:43.119339    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79kv\" (UniqueName: \"kubernetes.io/projected/e829788a-6c14-4fa6-bce5-6843972f60ce-kube-api-access-v79kv\") pod \"test-curl-8023-64a496\" (UID: \"e829788a-6c14-4fa6-bce5-6843972f60ce\") " pod="opendatahub/test-curl-8023-64a496"
Sep 12 01:10:43.220078 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:43.220047    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79kv\" (UniqueName: \"kubernetes.io/projected/e829788a-6c14-4fa6-bce5-6843972f60ce-kube-api-access-v79kv\") pod \"test-curl-8023-64a496\" (UID: \"e829788a-6c14-4fa6-bce5-6843972f60ce\") " pod="opendatahub/test-curl-8023-64a496"
Sep 12 01:10:43.228542 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:43.228496    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79kv\" (UniqueName: \"kubernetes.io/projected/e829788a-6c14-4fa6-bce5-6843972f60ce-kube-api-access-v79kv\") pod \"test-curl-8023-64a496\" (UID: \"e829788a-6c14-4fa6-bce5-6843972f60ce\") " pod="opendatahub/test-curl-8023-64a496"
Sep 12 01:10:43.294196 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:43.294164    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8023-64a496"
Sep 12 01:10:43.566583 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:43.566560    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8023-64a496"]
Sep 12 01:10:43.568912 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:10:43.568871    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode829788a_6c14_4fa6_bce5_6843972f60ce.slice/crio-298db19a76f57e08ed75759876e8dd519e9a27bcc2bcb94ff8130d8e051f5050 WatchSource:0}: Error finding container 298db19a76f57e08ed75759876e8dd519e9a27bcc2bcb94ff8130d8e051f5050: Status 404 returned error can't find the container with id 298db19a76f57e08ed75759876e8dd519e9a27bcc2bcb94ff8130d8e051f5050
Sep 12 01:10:44.161846 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:44.161807    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8023-64a496" event={"ID":"e829788a-6c14-4fa6-bce5-6843972f60ce","Type":"ContainerStarted","Data":"cca81c73cf076abc7cf1fe66a483a206b789ebbf2095243c45bbfbae6bac631d"}
Sep 12 01:10:44.161846 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:44.161853    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8023-64a496" event={"ID":"e829788a-6c14-4fa6-bce5-6843972f60ce","Type":"ContainerStarted","Data":"298db19a76f57e08ed75759876e8dd519e9a27bcc2bcb94ff8130d8e051f5050"}
Sep 12 01:10:44.177640 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:44.177595    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8023-64a496" podStartSLOduration=2.177580059 podStartE2EDuration="2.177580059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:10:44.176493713 +0000 UTC m=+1648.736945382" watchObservedRunningTime="2026-09-12 01:10:44.177580059 +0000 UTC m=+1648.738031729"
Sep 12 01:10:44.537702 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:44.537618    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:44.778919 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:44.778884    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8023-64a496"]
Sep 12 01:10:44.784096 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:44.784058    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8023-64a496"]
Sep 12 01:10:45.166930 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:10:45.166900    2567 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8023-64a496" container="test-curl-8023-64a496"
Sep 12 01:10:45.234233 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:45.234203    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:45.833270 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:45.833235    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:45.964249 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:45.964219    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:46.101371 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:46.101279    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:46.171247 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:46.171217    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8023-64a496" podUID="e829788a-6c14-4fa6-bce5-6843972f60ce" containerName="test-curl-8023-64a496" containerID="cri-o://cca81c73cf076abc7cf1fe66a483a206b789ebbf2095243c45bbfbae6bac631d" gracePeriod=2
Sep 12 01:10:46.359687 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:46.359605    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:47.564336 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:47.564307    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:10:48.361465 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:48.361438    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8023-64a496"
Sep 12 01:10:48.367288 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:48.367262    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v79kv\" (UniqueName: \"kubernetes.io/projected/e829788a-6c14-4fa6-bce5-6843972f60ce-kube-api-access-v79kv\") pod \"e829788a-6c14-4fa6-bce5-6843972f60ce\" (UID: \"e829788a-6c14-4fa6-bce5-6843972f60ce\") "
Sep 12 01:10:48.369358 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:48.369337    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e829788a-6c14-4fa6-bce5-6843972f60ce-kube-api-access-v79kv" (OuterVolumeSpecName: "kube-api-access-v79kv") pod "e829788a-6c14-4fa6-bce5-6843972f60ce" (UID: "e829788a-6c14-4fa6-bce5-6843972f60ce"). InnerVolumeSpecName "kube-api-access-v79kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:10:48.468116 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:48.468030    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v79kv\" (UniqueName: \"kubernetes.io/projected/e829788a-6c14-4fa6-bce5-6843972f60ce-kube-api-access-v79kv\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:49.187421 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:49.187386    2567 generic.go:356] "Generic (PLEG): container finished" podID="e829788a-6c14-4fa6-bce5-6843972f60ce" containerID="cca81c73cf076abc7cf1fe66a483a206b789ebbf2095243c45bbfbae6bac631d" exitCode=137
Sep 12 01:10:49.187852 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:49.187436    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8023-64a496"
Sep 12 01:10:49.187852 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:49.187480    2567 scope.go:117] "RemoveContainer" containerID="cca81c73cf076abc7cf1fe66a483a206b789ebbf2095243c45bbfbae6bac631d"
Sep 12 01:10:49.196170 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:49.196150    2567 scope.go:117] "RemoveContainer" containerID="cca81c73cf076abc7cf1fe66a483a206b789ebbf2095243c45bbfbae6bac631d"
Sep 12 01:10:49.196446 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:10:49.196428    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cca81c73cf076abc7cf1fe66a483a206b789ebbf2095243c45bbfbae6bac631d\": container with ID starting with cca81c73cf076abc7cf1fe66a483a206b789ebbf2095243c45bbfbae6bac631d not found: ID does not exist" containerID="cca81c73cf076abc7cf1fe66a483a206b789ebbf2095243c45bbfbae6bac631d"
Sep 12 01:10:49.196505 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:49.196454    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cca81c73cf076abc7cf1fe66a483a206b789ebbf2095243c45bbfbae6bac631d"} err="failed to get container status \"cca81c73cf076abc7cf1fe66a483a206b789ebbf2095243c45bbfbae6bac631d\": rpc error: code = NotFound desc = could not find container \"cca81c73cf076abc7cf1fe66a483a206b789ebbf2095243c45bbfbae6bac631d\": container with ID starting with cca81c73cf076abc7cf1fe66a483a206b789ebbf2095243c45bbfbae6bac631d not found: ID does not exist"
Sep 12 01:10:49.197585 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:10:49.197566    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8023-64a496\" is forbidden: User \"system:node:ip-10-0-139-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-121.ec2.internal' and this object" podUID="e829788a-6c14-4fa6-bce5-6843972f60ce" pod="opendatahub/test-curl-8023-64a496"
Sep 12 01:10:49.973635 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:49.973601    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e829788a-6c14-4fa6-bce5-6843972f60ce" path="/var/lib/kubelet/pods/e829788a-6c14-4fa6-bce5-6843972f60ce/volumes"
Sep 12 01:10:52.027073 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:10:52.027041    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:11:00.627205 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.627176    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"]
Sep 12 01:11:00.627605 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.627531    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e829788a-6c14-4fa6-bce5-6843972f60ce" containerName="test-curl-8023-64a496"
Sep 12 01:11:00.627605 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.627543    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e829788a-6c14-4fa6-bce5-6843972f60ce" containerName="test-curl-8023-64a496"
Sep 12 01:11:00.627681 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.627626    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e829788a-6c14-4fa6-bce5-6843972f60ce" containerName="test-curl-8023-64a496"
Sep 12 01:11:00.632076 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.632052    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.635659 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.635633    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-61f73f\"/\"default-dockercfg-c4brc\""
Sep 12 01:11:00.635785 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.635668    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-61f73f\"/\"kube-root-ca.crt\""
Sep 12 01:11:00.635785 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.635637    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-61f73f\"/\"sub-test-model-61f73f-kserve-self-signed-certs\""
Sep 12 01:11:00.635785 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.635637    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-61f73f\"/\"openshift-service-ca.crt\""
Sep 12 01:11:00.641934 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.641912    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"]
Sep 12 01:11:00.660379 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.660352    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:11:00.674140 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.674116    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmgft\" (UniqueName: \"kubernetes.io/projected/ae048b21-8041-41a0-be6a-d20354331540-kube-api-access-vmgft\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.674241 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.674150    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-dshm\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.674241 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.674167    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae048b21-8041-41a0-be6a-d20354331540-tls-certs\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.674241 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.674193    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-model-cache\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.674241 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.674231    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-tmp-dir\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.674383 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.674364    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-home\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.775132 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.775098    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-model-cache\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.775132 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.775132    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-tmp-dir\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.775322 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.775201    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-home\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.775322 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.775244    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmgft\" (UniqueName: \"kubernetes.io/projected/ae048b21-8041-41a0-be6a-d20354331540-kube-api-access-vmgft\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.775427 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.775384    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-dshm\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.775485 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.775424    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae048b21-8041-41a0-be6a-d20354331540-tls-certs\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.775485 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.775475    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-model-cache\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.775610 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.775550    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-tmp-dir\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.775610 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.775594    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-home\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.777639 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.777613    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-dshm\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.778049 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.778029    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae048b21-8041-41a0-be6a-d20354331540-tls-certs\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.784062 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.784037    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmgft\" (UniqueName: \"kubernetes.io/projected/ae048b21-8041-41a0-be6a-d20354331540-kube-api-access-vmgft\") pod \"sub-test-model-61f73f-kserve-7959b96c89-skxxf\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:00.942276 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:00.942243    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:01.214561 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:01.214534    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"]
Sep 12 01:11:01.216265 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:11:01.216226    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae048b21_8041_41a0_be6a_d20354331540.slice/crio-ff54ee3bf2b79a5b6f6dce4275c26585f68bfbed971a063a1ca258e690c38381 WatchSource:0}: Error finding container ff54ee3bf2b79a5b6f6dce4275c26585f68bfbed971a063a1ca258e690c38381: Status 404 returned error can't find the container with id ff54ee3bf2b79a5b6f6dce4275c26585f68bfbed971a063a1ca258e690c38381
Sep 12 01:11:01.248166 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:01.248139    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf" event={"ID":"ae048b21-8041-41a0-be6a-d20354331540","Type":"ContainerStarted","Data":"ff54ee3bf2b79a5b6f6dce4275c26585f68bfbed971a063a1ca258e690c38381"}
Sep 12 01:11:02.254503 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:02.254462    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf" event={"ID":"ae048b21-8041-41a0-be6a-d20354331540","Type":"ContainerStarted","Data":"54848cb86c3884da1e5fcb57e73dfa8bfaa8651b127a9c3574e1bb6af41d0fda"}
Sep 12 01:11:02.272546 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:02.272501    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf" podStartSLOduration=2.27248634 podStartE2EDuration="2.27248634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:11:02.271378867 +0000 UTC m=+1666.831830555" watchObservedRunningTime="2026-09-12 01:11:02.27248634 +0000 UTC m=+1666.832938009"
Sep 12 01:11:10.942795 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:10.942759    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:10.943207 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:10.942814    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:10.955318 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:10.955291    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:11.307907 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:11.307825    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:11:30.898969 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:11:30.898914    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:12:07.387739 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:07.387706    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8023-597b9f"]
Sep 12 01:12:07.391167 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:07.391149    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8023-597b9f"
Sep 12 01:12:07.393462 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:07.393441    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n87bf\""
Sep 12 01:12:07.397449 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:07.397429    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8023-597b9f"]
Sep 12 01:12:07.496081 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:07.496050    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwq7s\" (UniqueName: \"kubernetes.io/projected/789c6eb8-e238-4b51-a14d-90e46baa81d6-kube-api-access-gwq7s\") pod \"test-curl-8023-597b9f\" (UID: \"789c6eb8-e238-4b51-a14d-90e46baa81d6\") " pod="opendatahub/test-curl-8023-597b9f"
Sep 12 01:12:07.596538 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:07.596491    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwq7s\" (UniqueName: \"kubernetes.io/projected/789c6eb8-e238-4b51-a14d-90e46baa81d6-kube-api-access-gwq7s\") pod \"test-curl-8023-597b9f\" (UID: \"789c6eb8-e238-4b51-a14d-90e46baa81d6\") " pod="opendatahub/test-curl-8023-597b9f"
Sep 12 01:12:07.607038 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:07.607010    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwq7s\" (UniqueName: \"kubernetes.io/projected/789c6eb8-e238-4b51-a14d-90e46baa81d6-kube-api-access-gwq7s\") pod \"test-curl-8023-597b9f\" (UID: \"789c6eb8-e238-4b51-a14d-90e46baa81d6\") " pod="opendatahub/test-curl-8023-597b9f"
Sep 12 01:12:07.700669 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:07.700635    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8023-597b9f"
Sep 12 01:12:07.970691 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:12:07.970665    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod789c6eb8_e238_4b51_a14d_90e46baa81d6.slice/crio-fdd797c26418cfb8ce924b5c051741cf54cbc45c930bdd326733e8b547521196 WatchSource:0}: Error finding container fdd797c26418cfb8ce924b5c051741cf54cbc45c930bdd326733e8b547521196: Status 404 returned error can't find the container with id fdd797c26418cfb8ce924b5c051741cf54cbc45c930bdd326733e8b547521196
Sep 12 01:12:07.974405 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:07.974383    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8023-597b9f"]
Sep 12 01:12:08.564842 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:08.564806    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8023-597b9f" event={"ID":"789c6eb8-e238-4b51-a14d-90e46baa81d6","Type":"ContainerStarted","Data":"9b6ed11e5c35298892601a80cdfff279d89e6e640146a6339cdb2cfb89f679a0"}
Sep 12 01:12:08.565215 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:08.564851    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8023-597b9f" event={"ID":"789c6eb8-e238-4b51-a14d-90e46baa81d6","Type":"ContainerStarted","Data":"fdd797c26418cfb8ce924b5c051741cf54cbc45c930bdd326733e8b547521196"}
Sep 12 01:12:08.580848 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:08.580805    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8023-597b9f" podStartSLOduration=1.580791466 podStartE2EDuration="1.580791466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:12:08.579281879 +0000 UTC m=+1733.139733563" watchObservedRunningTime="2026-09-12 01:12:08.580791466 +0000 UTC m=+1733.141243134"
Sep 12 01:12:09.171740 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:09.171705    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8023-597b9f"]
Sep 12 01:12:09.176320 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:09.176291    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8023-597b9f"]
Sep 12 01:12:09.281862 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:09.281775    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:12:09.297002 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:09.296974    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8023-b16fbd"]
Sep 12 01:12:09.297361 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:09.297348    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="789c6eb8-e238-4b51-a14d-90e46baa81d6" containerName="test-curl-8023-597b9f"
Sep 12 01:12:09.297408 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:09.297363    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="789c6eb8-e238-4b51-a14d-90e46baa81d6" containerName="test-curl-8023-597b9f"
Sep 12 01:12:09.297445 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:09.297430    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="789c6eb8-e238-4b51-a14d-90e46baa81d6" containerName="test-curl-8023-597b9f"
Sep 12 01:12:09.300496 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:09.300482    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8023-b16fbd"
Sep 12 01:12:09.304426 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:09.304403    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8023-b16fbd"]
Sep 12 01:12:09.415311 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:09.415271    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk4qj\" (UniqueName: \"kubernetes.io/projected/f0e17562-16ad-48ba-b964-6a5fed6d3722-kube-api-access-xk4qj\") pod \"test-curl-8023-b16fbd\" (UID: \"f0e17562-16ad-48ba-b964-6a5fed6d3722\") " pod="opendatahub/test-curl-8023-b16fbd"
Sep 12 01:12:09.516177 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:09.516140    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk4qj\" (UniqueName: \"kubernetes.io/projected/f0e17562-16ad-48ba-b964-6a5fed6d3722-kube-api-access-xk4qj\") pod \"test-curl-8023-b16fbd\" (UID: \"f0e17562-16ad-48ba-b964-6a5fed6d3722\") " pod="opendatahub/test-curl-8023-b16fbd"
Sep 12 01:12:09.524995 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:09.524936    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk4qj\" (UniqueName: \"kubernetes.io/projected/f0e17562-16ad-48ba-b964-6a5fed6d3722-kube-api-access-xk4qj\") pod \"test-curl-8023-b16fbd\" (UID: \"f0e17562-16ad-48ba-b964-6a5fed6d3722\") " pod="opendatahub/test-curl-8023-b16fbd"
Sep 12 01:12:09.570680 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:12:09.570598    2567 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8023-597b9f" container="test-curl-8023-597b9f"
Sep 12 01:12:09.610785 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:09.610754    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8023-b16fbd"
Sep 12 01:12:09.878774 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:09.878749    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8023-b16fbd"]
Sep 12 01:12:09.880664 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:12:09.880640    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0e17562_16ad_48ba_b964_6a5fed6d3722.slice/crio-248f75ccff170ad4dfd71ac118f21cdfb191e9600afb029afda13def891657a4 WatchSource:0}: Error finding container 248f75ccff170ad4dfd71ac118f21cdfb191e9600afb029afda13def891657a4: Status 404 returned error can't find the container with id 248f75ccff170ad4dfd71ac118f21cdfb191e9600afb029afda13def891657a4
Sep 12 01:12:10.577046 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:10.577011    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8023-b16fbd" event={"ID":"f0e17562-16ad-48ba-b964-6a5fed6d3722","Type":"ContainerStarted","Data":"c50305adb277b26d3f5e80701fc137fdb782d100242f849255102a5b26dcd9e2"}
Sep 12 01:12:10.577046 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:10.577056    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8023-b16fbd" event={"ID":"f0e17562-16ad-48ba-b964-6a5fed6d3722","Type":"ContainerStarted","Data":"248f75ccff170ad4dfd71ac118f21cdfb191e9600afb029afda13def891657a4"}
Sep 12 01:12:10.577521 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:10.577113    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8023-597b9f" podUID="789c6eb8-e238-4b51-a14d-90e46baa81d6" containerName="test-curl-8023-597b9f" containerID="cri-o://9b6ed11e5c35298892601a80cdfff279d89e6e640146a6339cdb2cfb89f679a0" gracePeriod=2
Sep 12 01:12:10.594308 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:10.594263    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8023-b16fbd" podStartSLOduration=1.5942509390000001 podStartE2EDuration="1.594250939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:12:10.593238923 +0000 UTC m=+1735.153690618" watchObservedRunningTime="2026-09-12 01:12:10.594250939 +0000 UTC m=+1735.154702606"
Sep 12 01:12:11.170998 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:11.170962    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8023-b16fbd"]
Sep 12 01:12:11.175155 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:11.175120    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8023-b16fbd"]
Sep 12 01:12:11.581734 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:12:11.581699    2567 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8023-b16fbd" container="test-curl-8023-b16fbd"
Sep 12 01:12:12.587033 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:12.587001    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8023-b16fbd" podUID="f0e17562-16ad-48ba-b964-6a5fed6d3722" containerName="test-curl-8023-b16fbd" containerID="cri-o://c50305adb277b26d3f5e80701fc137fdb782d100242f849255102a5b26dcd9e2" gracePeriod=2
Sep 12 01:12:12.758127 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:12.758100    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8023-597b9f"
Sep 12 01:12:12.849532 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:12.849443    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwq7s\" (UniqueName: \"kubernetes.io/projected/789c6eb8-e238-4b51-a14d-90e46baa81d6-kube-api-access-gwq7s\") pod \"789c6eb8-e238-4b51-a14d-90e46baa81d6\" (UID: \"789c6eb8-e238-4b51-a14d-90e46baa81d6\") "
Sep 12 01:12:12.851690 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:12.851665    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789c6eb8-e238-4b51-a14d-90e46baa81d6-kube-api-access-gwq7s" (OuterVolumeSpecName: "kube-api-access-gwq7s") pod "789c6eb8-e238-4b51-a14d-90e46baa81d6" (UID: "789c6eb8-e238-4b51-a14d-90e46baa81d6"). InnerVolumeSpecName "kube-api-access-gwq7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:12:12.950572 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:12.950537    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwq7s\" (UniqueName: \"kubernetes.io/projected/789c6eb8-e238-4b51-a14d-90e46baa81d6-kube-api-access-gwq7s\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:12:13.170942 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:13.170906    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"]
Sep 12 01:12:13.195700 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:13.195667    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:12:13.592519 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:13.592416    2567 generic.go:356] "Generic (PLEG): container finished" podID="789c6eb8-e238-4b51-a14d-90e46baa81d6" containerID="9b6ed11e5c35298892601a80cdfff279d89e6e640146a6339cdb2cfb89f679a0" exitCode=137
Sep 12 01:12:13.592519 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:13.592463    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8023-597b9f"
Sep 12 01:12:13.593012 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:13.592523    2567 scope.go:117] "RemoveContainer" containerID="9b6ed11e5c35298892601a80cdfff279d89e6e640146a6339cdb2cfb89f679a0"
Sep 12 01:12:13.601419 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:13.601401    2567 scope.go:117] "RemoveContainer" containerID="9b6ed11e5c35298892601a80cdfff279d89e6e640146a6339cdb2cfb89f679a0"
Sep 12 01:12:13.601701 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:12:13.601673    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6ed11e5c35298892601a80cdfff279d89e6e640146a6339cdb2cfb89f679a0\": container with ID starting with 9b6ed11e5c35298892601a80cdfff279d89e6e640146a6339cdb2cfb89f679a0 not found: ID does not exist" containerID="9b6ed11e5c35298892601a80cdfff279d89e6e640146a6339cdb2cfb89f679a0"
Sep 12 01:12:13.601746 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:13.601711    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6ed11e5c35298892601a80cdfff279d89e6e640146a6339cdb2cfb89f679a0"} err="failed to get container status \"9b6ed11e5c35298892601a80cdfff279d89e6e640146a6339cdb2cfb89f679a0\": rpc error: code = NotFound desc = could not find container \"9b6ed11e5c35298892601a80cdfff279d89e6e640146a6339cdb2cfb89f679a0\": container with ID starting with 9b6ed11e5c35298892601a80cdfff279d89e6e640146a6339cdb2cfb89f679a0 not found: ID does not exist"
Sep 12 01:12:13.602642 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:12:13.602620    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8023-597b9f\" is forbidden: User \"system:node:ip-10-0-139-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-121.ec2.internal' and this object" podUID="789c6eb8-e238-4b51-a14d-90e46baa81d6" pod="opendatahub/test-curl-8023-597b9f"
Sep 12 01:12:13.973125 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:13.973098    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="789c6eb8-e238-4b51-a14d-90e46baa81d6" path="/var/lib/kubelet/pods/789c6eb8-e238-4b51-a14d-90e46baa81d6/volumes"
Sep 12 01:12:14.172775 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:14.172744    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:12:14.769527 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:14.769502    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8023-b16fbd"
Sep 12 01:12:14.863521 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:14.863445    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk4qj\" (UniqueName: \"kubernetes.io/projected/f0e17562-16ad-48ba-b964-6a5fed6d3722-kube-api-access-xk4qj\") pod \"f0e17562-16ad-48ba-b964-6a5fed6d3722\" (UID: \"f0e17562-16ad-48ba-b964-6a5fed6d3722\") "
Sep 12 01:12:14.865621 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:14.865597    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e17562-16ad-48ba-b964-6a5fed6d3722-kube-api-access-xk4qj" (OuterVolumeSpecName: "kube-api-access-xk4qj") pod "f0e17562-16ad-48ba-b964-6a5fed6d3722" (UID: "f0e17562-16ad-48ba-b964-6a5fed6d3722"). InnerVolumeSpecName "kube-api-access-xk4qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:12:14.964167 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:14.964142    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk4qj\" (UniqueName: \"kubernetes.io/projected/f0e17562-16ad-48ba-b964-6a5fed6d3722-kube-api-access-xk4qj\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:12:15.604673 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:15.604605    2567 generic.go:356] "Generic (PLEG): container finished" podID="f0e17562-16ad-48ba-b964-6a5fed6d3722" containerID="c50305adb277b26d3f5e80701fc137fdb782d100242f849255102a5b26dcd9e2" exitCode=137
Sep 12 01:12:15.604869 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:15.604698    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8023-b16fbd"
Sep 12 01:12:15.604869 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:15.604752    2567 scope.go:117] "RemoveContainer" containerID="c50305adb277b26d3f5e80701fc137fdb782d100242f849255102a5b26dcd9e2"
Sep 12 01:12:15.613821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:15.613804    2567 scope.go:117] "RemoveContainer" containerID="c50305adb277b26d3f5e80701fc137fdb782d100242f849255102a5b26dcd9e2"
Sep 12 01:12:15.614207 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:12:15.614179    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50305adb277b26d3f5e80701fc137fdb782d100242f849255102a5b26dcd9e2\": container with ID starting with c50305adb277b26d3f5e80701fc137fdb782d100242f849255102a5b26dcd9e2 not found: ID does not exist" containerID="c50305adb277b26d3f5e80701fc137fdb782d100242f849255102a5b26dcd9e2"
Sep 12 01:12:15.614283 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:15.614215    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50305adb277b26d3f5e80701fc137fdb782d100242f849255102a5b26dcd9e2"} err="failed to get container status \"c50305adb277b26d3f5e80701fc137fdb782d100242f849255102a5b26dcd9e2\": rpc error: code = NotFound desc = could not find container \"c50305adb277b26d3f5e80701fc137fdb782d100242f849255102a5b26dcd9e2\": container with ID starting with c50305adb277b26d3f5e80701fc137fdb782d100242f849255102a5b26dcd9e2 not found: ID does not exist"
Sep 12 01:12:15.616047 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:12:15.616026    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8023-b16fbd\" is forbidden: User \"system:node:ip-10-0-139-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-121.ec2.internal' and this object" podUID="f0e17562-16ad-48ba-b964-6a5fed6d3722" pod="opendatahub/test-curl-8023-b16fbd"
Sep 12 01:12:15.972341 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:12:15.972302    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8023-b16fbd\" is forbidden: User \"system:node:ip-10-0-139-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-121.ec2.internal' and this object" podUID="f0e17562-16ad-48ba-b964-6a5fed6d3722" pod="opendatahub/test-curl-8023-b16fbd"
Sep 12 01:12:15.973646 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:15.973627    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0e17562-16ad-48ba-b964-6a5fed6d3722" path="/var/lib/kubelet/pods/f0e17562-16ad-48ba-b964-6a5fed6d3722/volumes"
Sep 12 01:12:19.187158 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:19.187122    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:12:19.288775 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:19.288747    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:12:28.190583 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.190538    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf" podUID="ae048b21-8041-41a0-be6a-d20354331540" containerName="main" containerID="cri-o://54848cb86c3884da1e5fcb57e73dfa8bfaa8651b127a9c3574e1bb6af41d0fda" gracePeriod=45
Sep 12 01:12:28.440838 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.440772    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:12:28.492538 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.492503    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-home\") pod \"ae048b21-8041-41a0-be6a-d20354331540\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") "
Sep 12 01:12:28.492538 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.492549    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-dshm\") pod \"ae048b21-8041-41a0-be6a-d20354331540\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") "
Sep 12 01:12:28.492783 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.492599    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-tmp-dir\") pod \"ae048b21-8041-41a0-be6a-d20354331540\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") "
Sep 12 01:12:28.492783 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.492620    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae048b21-8041-41a0-be6a-d20354331540-tls-certs\") pod \"ae048b21-8041-41a0-be6a-d20354331540\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") "
Sep 12 01:12:28.492783 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.492664    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-model-cache\") pod \"ae048b21-8041-41a0-be6a-d20354331540\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") "
Sep 12 01:12:28.492978 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.492846    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-home" (OuterVolumeSpecName: "home") pod "ae048b21-8041-41a0-be6a-d20354331540" (UID: "ae048b21-8041-41a0-be6a-d20354331540"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:12:28.492978 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.492883    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmgft\" (UniqueName: \"kubernetes.io/projected/ae048b21-8041-41a0-be6a-d20354331540-kube-api-access-vmgft\") pod \"ae048b21-8041-41a0-be6a-d20354331540\" (UID: \"ae048b21-8041-41a0-be6a-d20354331540\") "
Sep 12 01:12:28.492978 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.492913    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae048b21-8041-41a0-be6a-d20354331540" (UID: "ae048b21-8041-41a0-be6a-d20354331540"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:12:28.492978 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.492931    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae048b21-8041-41a0-be6a-d20354331540" (UID: "ae048b21-8041-41a0-be6a-d20354331540"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:12:28.493303 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.493238    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-model-cache\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:12:28.493303 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.493260    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-home\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:12:28.493303 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.493271    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-tmp-dir\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:12:28.495035 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.495009    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-dshm" (OuterVolumeSpecName: "dshm") pod "ae048b21-8041-41a0-be6a-d20354331540" (UID: "ae048b21-8041-41a0-be6a-d20354331540"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:12:28.495133 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.495034    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae048b21-8041-41a0-be6a-d20354331540-kube-api-access-vmgft" (OuterVolumeSpecName: "kube-api-access-vmgft") pod "ae048b21-8041-41a0-be6a-d20354331540" (UID: "ae048b21-8041-41a0-be6a-d20354331540"). InnerVolumeSpecName "kube-api-access-vmgft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:12:28.495133 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.495063    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae048b21-8041-41a0-be6a-d20354331540-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae048b21-8041-41a0-be6a-d20354331540" (UID: "ae048b21-8041-41a0-be6a-d20354331540"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:12:28.593877 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.593826    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae048b21-8041-41a0-be6a-d20354331540-tls-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:12:28.593877 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.593873    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmgft\" (UniqueName: \"kubernetes.io/projected/ae048b21-8041-41a0-be6a-d20354331540-kube-api-access-vmgft\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:12:28.593877 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.593884    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae048b21-8041-41a0-be6a-d20354331540-dshm\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:12:28.672804 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.672768    2567 generic.go:356] "Generic (PLEG): container finished" podID="ae048b21-8041-41a0-be6a-d20354331540" containerID="54848cb86c3884da1e5fcb57e73dfa8bfaa8651b127a9c3574e1bb6af41d0fda" exitCode=0
Sep 12 01:12:28.673002 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.672812    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"
Sep 12 01:12:28.673002 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.672848    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf" event={"ID":"ae048b21-8041-41a0-be6a-d20354331540","Type":"ContainerDied","Data":"54848cb86c3884da1e5fcb57e73dfa8bfaa8651b127a9c3574e1bb6af41d0fda"}
Sep 12 01:12:28.673002 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.672891    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf" event={"ID":"ae048b21-8041-41a0-be6a-d20354331540","Type":"ContainerDied","Data":"ff54ee3bf2b79a5b6f6dce4275c26585f68bfbed971a063a1ca258e690c38381"}
Sep 12 01:12:28.673002 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.672910    2567 scope.go:117] "RemoveContainer" containerID="54848cb86c3884da1e5fcb57e73dfa8bfaa8651b127a9c3574e1bb6af41d0fda"
Sep 12 01:12:28.681720 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.681705    2567 scope.go:117] "RemoveContainer" containerID="54848cb86c3884da1e5fcb57e73dfa8bfaa8651b127a9c3574e1bb6af41d0fda"
Sep 12 01:12:28.682006 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:12:28.681989    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54848cb86c3884da1e5fcb57e73dfa8bfaa8651b127a9c3574e1bb6af41d0fda\": container with ID starting with 54848cb86c3884da1e5fcb57e73dfa8bfaa8651b127a9c3574e1bb6af41d0fda not found: ID does not exist" containerID="54848cb86c3884da1e5fcb57e73dfa8bfaa8651b127a9c3574e1bb6af41d0fda"
Sep 12 01:12:28.682064 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.682019    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54848cb86c3884da1e5fcb57e73dfa8bfaa8651b127a9c3574e1bb6af41d0fda"} err="failed to get container status \"54848cb86c3884da1e5fcb57e73dfa8bfaa8651b127a9c3574e1bb6af41d0fda\": rpc error: code = NotFound desc = could not find container \"54848cb86c3884da1e5fcb57e73dfa8bfaa8651b127a9c3574e1bb6af41d0fda\": container with ID starting with 54848cb86c3884da1e5fcb57e73dfa8bfaa8651b127a9c3574e1bb6af41d0fda not found: ID does not exist"
Sep 12 01:12:28.692884 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.692834    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"]
Sep 12 01:12:28.698801 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:28.698775    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/sub-test-model-61f73f-kserve-7959b96c89-skxxf"]
Sep 12 01:12:29.973756 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:29.973723    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae048b21-8041-41a0-be6a-d20354331540" path="/var/lib/kubelet/pods/ae048b21-8041-41a0-be6a-d20354331540/volumes"
Sep 12 01:12:36.355080 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.355046    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"]
Sep 12 01:12:36.355645 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.355615    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae048b21-8041-41a0-be6a-d20354331540" containerName="main"
Sep 12 01:12:36.355645 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.355641    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae048b21-8041-41a0-be6a-d20354331540" containerName="main"
Sep 12 01:12:36.355813 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.355654    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e17562-16ad-48ba-b964-6a5fed6d3722" containerName="test-curl-8023-b16fbd"
Sep 12 01:12:36.355813 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.355660    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e17562-16ad-48ba-b964-6a5fed6d3722" containerName="test-curl-8023-b16fbd"
Sep 12 01:12:36.355813 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.355728    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae048b21-8041-41a0-be6a-d20354331540" containerName="main"
Sep 12 01:12:36.355813 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.355740    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e17562-16ad-48ba-b964-6a5fed6d3722" containerName="test-curl-8023-b16fbd"
Sep 12 01:12:36.360236 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.360213    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.363772 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.363742    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-61f73f\"/\"default-dockercfg-c4brc\""
Sep 12 01:12:36.363915 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.363779    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-61f73f\"/\"kube-root-ca.crt\""
Sep 12 01:12:36.363915 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.363779    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-61f73f\"/\"openshift-service-ca.crt\""
Sep 12 01:12:36.363915 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.363742    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-61f73f\"/\"rate-test-model-61f73f-kserve-self-signed-certs\""
Sep 12 01:12:36.368387 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.368367    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"]
Sep 12 01:12:36.382533 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.382503    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:12:36.465971 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.465906    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f5d2f7-151a-4308-9f48-180f0092e9f9-tls-certs\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.466147 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.465990    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmk6f\" (UniqueName: \"kubernetes.io/projected/22f5d2f7-151a-4308-9f48-180f0092e9f9-kube-api-access-jmk6f\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.466147 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.466046    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-dshm\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.466147 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.466078    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-tmp-dir\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.466285 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.466181    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-model-cache\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.466285 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.466219    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-home\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.567384 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.567347    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-tmp-dir\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.567597 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.567427    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-model-cache\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.567597 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.567445    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-home\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.567597 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.567462    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f5d2f7-151a-4308-9f48-180f0092e9f9-tls-certs\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.567597 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.567490    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmk6f\" (UniqueName: \"kubernetes.io/projected/22f5d2f7-151a-4308-9f48-180f0092e9f9-kube-api-access-jmk6f\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.567597 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.567565    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-dshm\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.567875 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.567801    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-tmp-dir\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.567941 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.567906    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-home\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.568055 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.568032    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-model-cache\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.569929 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.569904    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-dshm\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.570206 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.570189    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f5d2f7-151a-4308-9f48-180f0092e9f9-tls-certs\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.578422 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.578400    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmk6f\" (UniqueName: \"kubernetes.io/projected/22f5d2f7-151a-4308-9f48-180f0092e9f9-kube-api-access-jmk6f\") pod \"rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.670542 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.670503    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:36.949041 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:36.948850    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"]
Sep 12 01:12:36.951892 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:12:36.951859    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22f5d2f7_151a_4308_9f48_180f0092e9f9.slice/crio-7ddbf366202aca8d250699aa813a4178cd44b587cd1941b088d2f7636bd7c892 WatchSource:0}: Error finding container 7ddbf366202aca8d250699aa813a4178cd44b587cd1941b088d2f7636bd7c892: Status 404 returned error can't find the container with id 7ddbf366202aca8d250699aa813a4178cd44b587cd1941b088d2f7636bd7c892
Sep 12 01:12:37.719685 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:37.719650    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq" event={"ID":"22f5d2f7-151a-4308-9f48-180f0092e9f9","Type":"ContainerStarted","Data":"5909ed19716217354d23d108b914379f5473db3c354fb00c6b21c75da5b360dd"}
Sep 12 01:12:37.719685 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:37.719689    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq" event={"ID":"22f5d2f7-151a-4308-9f48-180f0092e9f9","Type":"ContainerStarted","Data":"7ddbf366202aca8d250699aa813a4178cd44b587cd1941b088d2f7636bd7c892"}
Sep 12 01:12:37.742389 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:37.742341    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq" podStartSLOduration=1.7423256230000002 podStartE2EDuration="1.742325623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:12:37.740304478 +0000 UTC m=+1762.300756147" watchObservedRunningTime="2026-09-12 01:12:37.742325623 +0000 UTC m=+1762.302777290"
Sep 12 01:12:46.671212 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:46.671175    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:46.671715 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:46.671444    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:46.683925 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:46.683895    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:46.775436 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:46.775404    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:12:59.871068 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:12:59.871029    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:00.024263 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:00.024223    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:08.988035 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:08.987936    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:09.088971 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:09.088915    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:16.008797 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:16.008770    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 01:13:16.012181 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:16.012156    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 01:13:24.402274 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:24.402240    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:33.364242 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:33.364206    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"]
Sep 12 01:13:33.387002 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:33.386971    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:34.344782 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:34.344750    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:34.804602 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:34.804572    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:35.109012 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:35.108899    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:35.273403 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:35.273368    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:38.217136 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:38.217098    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:38.368177 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:38.368145    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:38.464570 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:38.464542    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:39.097244 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:39.097206    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:39.811357 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:39.811319    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"]
Sep 12 01:13:39.815581 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:39.815557    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:39.818017 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:39.817986    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5fbbcd-serving-cert\""
Sep 12 01:13:39.824979 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:39.824931    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"]
Sep 12 01:13:39.840651 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:39.840617    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:39.943670 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:39.943628    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54jbt\" (UniqueName: \"kubernetes.io/projected/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-kube-api-access-54jbt\") pod \"maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:39.943867 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:39.943707    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:39.943867 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:39.943728    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:39.943867 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:39.943760    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-metrics-certs\") pod \"maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:39.944393 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:39.944366    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:40.044388 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:40.044352    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54jbt\" (UniqueName: \"kubernetes.io/projected/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-kube-api-access-54jbt\") pod \"maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:40.044582 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:40.044423    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:40.044582 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:40.044452    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:40.044582 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:40.044495    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-metrics-certs\") pod \"maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:40.044582 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:13:40.044574    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-serving-cert: secret "maas-api-e2e-ipp-a-5fbbcd-serving-cert" not found
Sep 12 01:13:40.044810 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:13:40.044635    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-maas-api-tls podName:25fcaa1e-672c-4c6d-8577-ab57dcebdb21 nodeName:}" failed. No retries permitted until 2026-09-12 01:13:40.544619363 +0000 UTC m=+1825.105071009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-maas-api-tls") pod "maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs" (UID: "25fcaa1e-672c-4c6d-8577-ab57dcebdb21") : secret "maas-api-e2e-ipp-a-5fbbcd-serving-cert" not found
Sep 12 01:13:40.045112 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:40.045087    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:40.047225 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:40.047205    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-metrics-certs\") pod \"maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:40.053741 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:40.053715    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54jbt\" (UniqueName: \"kubernetes.io/projected/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-kube-api-access-54jbt\") pod \"maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:40.255362 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:40.255331    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:40.350935 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:40.350898    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:40.446125 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:40.446087    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:40.549066 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:40.548990    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:40.551418 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:40.551396    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:40.728748 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:40.728720    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:41.008899 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:41.008877    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"]
Sep 12 01:13:41.010729 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:13:41.010707    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fcaa1e_672c_4c6d_8577_ab57dcebdb21.slice/crio-6a689ac5cfc97788f1694558dfe1bbea07854ca8fcd8171a6aa409deed19644b WatchSource:0}: Error finding container 6a689ac5cfc97788f1694558dfe1bbea07854ca8fcd8171a6aa409deed19644b: Status 404 returned error can't find the container with id 6a689ac5cfc97788f1694558dfe1bbea07854ca8fcd8171a6aa409deed19644b
Sep 12 01:13:41.012077 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:41.012059    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 01:13:41.038276 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:41.038242    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs" event={"ID":"25fcaa1e-672c-4c6d-8577-ab57dcebdb21","Type":"ContainerStarted","Data":"6a689ac5cfc97788f1694558dfe1bbea07854ca8fcd8171a6aa409deed19644b"}
Sep 12 01:13:42.044240 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:42.044207    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs" event={"ID":"25fcaa1e-672c-4c6d-8577-ab57dcebdb21","Type":"ContainerStarted","Data":"bb20731a1ee65669d1a0f1308e4e2ba73038fc9e3fdff1a95d84d76fbcc95feb"}
Sep 12 01:13:42.044607 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:42.044321    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:42.063532 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:42.063485    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs" podStartSLOduration=2.560017894 podStartE2EDuration="3.063471388s" totalImagesPullingTime="503.453494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:13:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:13:42.061969514 +0000 UTC m=+1826.622421195" watchObservedRunningTime="2026-09-12 01:13:42.063471388 +0000 UTC m=+1826.623923056"
Sep 12 01:13:47.994204 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:47.994133    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:13:48.054356 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.054329    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:13:48.384239 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.384148    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq" podUID="22f5d2f7-151a-4308-9f48-180f0092e9f9" containerName="main" containerID="cri-o://5909ed19716217354d23d108b914379f5473db3c354fb00c6b21c75da5b360dd" gracePeriod=45
Sep 12 01:13:48.638933 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.638879    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:13:48.722657 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.722626    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-dshm\") pod \"22f5d2f7-151a-4308-9f48-180f0092e9f9\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") "
Sep 12 01:13:48.722825 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.722680    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmk6f\" (UniqueName: \"kubernetes.io/projected/22f5d2f7-151a-4308-9f48-180f0092e9f9-kube-api-access-jmk6f\") pod \"22f5d2f7-151a-4308-9f48-180f0092e9f9\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") "
Sep 12 01:13:48.722825 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.722707    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-tmp-dir\") pod \"22f5d2f7-151a-4308-9f48-180f0092e9f9\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") "
Sep 12 01:13:48.722825 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.722732    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-model-cache\") pod \"22f5d2f7-151a-4308-9f48-180f0092e9f9\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") "
Sep 12 01:13:48.722825 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.722790    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-home\") pod \"22f5d2f7-151a-4308-9f48-180f0092e9f9\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") "
Sep 12 01:13:48.723083 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.722828    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f5d2f7-151a-4308-9f48-180f0092e9f9-tls-certs\") pod \"22f5d2f7-151a-4308-9f48-180f0092e9f9\" (UID: \"22f5d2f7-151a-4308-9f48-180f0092e9f9\") "
Sep 12 01:13:48.723083 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.722979    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22f5d2f7-151a-4308-9f48-180f0092e9f9" (UID: "22f5d2f7-151a-4308-9f48-180f0092e9f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:13:48.723083 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.723027    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "22f5d2f7-151a-4308-9f48-180f0092e9f9" (UID: "22f5d2f7-151a-4308-9f48-180f0092e9f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:13:48.723255 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.723103    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-home" (OuterVolumeSpecName: "home") pod "22f5d2f7-151a-4308-9f48-180f0092e9f9" (UID: "22f5d2f7-151a-4308-9f48-180f0092e9f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:13:48.723255 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.723248    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-tmp-dir\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:13:48.723345 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.723262    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-model-cache\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:13:48.723345 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.723271    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-home\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:13:48.724879 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.724859    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-dshm" (OuterVolumeSpecName: "dshm") pod "22f5d2f7-151a-4308-9f48-180f0092e9f9" (UID: "22f5d2f7-151a-4308-9f48-180f0092e9f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:13:48.725123 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.725098    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f5d2f7-151a-4308-9f48-180f0092e9f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22f5d2f7-151a-4308-9f48-180f0092e9f9" (UID: "22f5d2f7-151a-4308-9f48-180f0092e9f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:13:48.725217 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.725103    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f5d2f7-151a-4308-9f48-180f0092e9f9-kube-api-access-jmk6f" (OuterVolumeSpecName: "kube-api-access-jmk6f") pod "22f5d2f7-151a-4308-9f48-180f0092e9f9" (UID: "22f5d2f7-151a-4308-9f48-180f0092e9f9"). InnerVolumeSpecName "kube-api-access-jmk6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:13:48.823848 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.823824    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22f5d2f7-151a-4308-9f48-180f0092e9f9-dshm\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:13:48.823848 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.823847    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmk6f\" (UniqueName: \"kubernetes.io/projected/22f5d2f7-151a-4308-9f48-180f0092e9f9-kube-api-access-jmk6f\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:13:48.824049 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:48.823858    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f5d2f7-151a-4308-9f48-180f0092e9f9-tls-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:13:49.077525 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:49.077496    2567 generic.go:356] "Generic (PLEG): container finished" podID="22f5d2f7-151a-4308-9f48-180f0092e9f9" containerID="5909ed19716217354d23d108b914379f5473db3c354fb00c6b21c75da5b360dd" exitCode=0
Sep 12 01:13:49.077982 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:49.077539    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"
Sep 12 01:13:49.077982 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:49.077582    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq" event={"ID":"22f5d2f7-151a-4308-9f48-180f0092e9f9","Type":"ContainerDied","Data":"5909ed19716217354d23d108b914379f5473db3c354fb00c6b21c75da5b360dd"}
Sep 12 01:13:49.077982 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:49.077625    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq" event={"ID":"22f5d2f7-151a-4308-9f48-180f0092e9f9","Type":"ContainerDied","Data":"7ddbf366202aca8d250699aa813a4178cd44b587cd1941b088d2f7636bd7c892"}
Sep 12 01:13:49.077982 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:49.077643    2567 scope.go:117] "RemoveContainer" containerID="5909ed19716217354d23d108b914379f5473db3c354fb00c6b21c75da5b360dd"
Sep 12 01:13:49.086842 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:49.086824    2567 scope.go:117] "RemoveContainer" containerID="5909ed19716217354d23d108b914379f5473db3c354fb00c6b21c75da5b360dd"
Sep 12 01:13:49.087142 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:13:49.087124    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5909ed19716217354d23d108b914379f5473db3c354fb00c6b21c75da5b360dd\": container with ID starting with 5909ed19716217354d23d108b914379f5473db3c354fb00c6b21c75da5b360dd not found: ID does not exist" containerID="5909ed19716217354d23d108b914379f5473db3c354fb00c6b21c75da5b360dd"
Sep 12 01:13:49.087196 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:49.087151    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5909ed19716217354d23d108b914379f5473db3c354fb00c6b21c75da5b360dd"} err="failed to get container status \"5909ed19716217354d23d108b914379f5473db3c354fb00c6b21c75da5b360dd\": rpc error: code = NotFound desc = could not find container \"5909ed19716217354d23d108b914379f5473db3c354fb00c6b21c75da5b360dd\": container with ID starting with 5909ed19716217354d23d108b914379f5473db3c354fb00c6b21c75da5b360dd not found: ID does not exist"
Sep 12 01:13:49.101266 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:49.101234    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"]
Sep 12 01:13:49.106737 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:49.106719    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/rate-test-model-61f73f-kserve-5dd5d9dd67-qx6nq"]
Sep 12 01:13:49.982557 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:13:49.982519    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22f5d2f7-151a-4308-9f48-180f0092e9f9" path="/var/lib/kubelet/pods/22f5d2f7-151a-4308-9f48-180f0092e9f9/volumes"
Sep 12 01:14:00.763440 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:00.763403    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:02.112798 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:02.112766    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:02.887074 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:02.887044    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:03.397788 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:03.397748    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:03.583878 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:03.583829    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:04.082549 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:04.082514    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:04.415908 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:04.415873    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:04.506934 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:04.506902    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:04.740048 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:04.739936    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:07.072885 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:07.072849    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:07.802185 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:07.802153    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:07.972272 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:07.972219    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:08.802643 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:08.802614    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:09.585786 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:09.585752    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:10.296515 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.296466    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"]
Sep 12 01:14:10.296916 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.296883    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f5d2f7-151a-4308-9f48-180f0092e9f9" containerName="main"
Sep 12 01:14:10.296916 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.296896    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f5d2f7-151a-4308-9f48-180f0092e9f9" containerName="main"
Sep 12 01:14:10.297035 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.297002    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f5d2f7-151a-4308-9f48-180f0092e9f9" containerName="main"
Sep 12 01:14:10.301860 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.301838    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:10.304327 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.304303    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-57b051-serving-cert\""
Sep 12 01:14:10.313551 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.313531    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"]
Sep 12 01:14:10.331555 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.331531    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:10.392912 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.392868    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:10.422124 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.422096    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fbd097-cd0a-4b4b-b06c-af8d956819a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:10.422227 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.422139    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8pt5\" (UniqueName: \"kubernetes.io/projected/39fbd097-cd0a-4b4b-b06c-af8d956819a9-kube-api-access-h8pt5\") pod \"maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:10.422227 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.422214    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39fbd097-cd0a-4b4b-b06c-af8d956819a9-metrics-certs\") pod \"maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:10.422302 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.422253    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39fbd097-cd0a-4b4b-b06c-af8d956819a9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:10.523381 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.523344    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39fbd097-cd0a-4b4b-b06c-af8d956819a9-metrics-certs\") pod \"maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:10.523524 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.523394    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39fbd097-cd0a-4b4b-b06c-af8d956819a9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:10.523524 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.523463    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fbd097-cd0a-4b4b-b06c-af8d956819a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:10.523524 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.523509    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8pt5\" (UniqueName: \"kubernetes.io/projected/39fbd097-cd0a-4b4b-b06c-af8d956819a9-kube-api-access-h8pt5\") pod \"maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:10.524209 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.524181    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fbd097-cd0a-4b4b-b06c-af8d956819a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:10.525897 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.525874    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39fbd097-cd0a-4b4b-b06c-af8d956819a9-metrics-certs\") pod \"maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:10.526006 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.525987    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39fbd097-cd0a-4b4b-b06c-af8d956819a9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:10.533461 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.533439    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8pt5\" (UniqueName: \"kubernetes.io/projected/39fbd097-cd0a-4b4b-b06c-af8d956819a9-kube-api-access-h8pt5\") pod \"maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:10.612884 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.612813    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:10.823806 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.823773    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:10.884918 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.884837    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"]
Sep 12 01:14:10.888100 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:14:10.888073    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fbd097_cd0a_4b4b_b06c_af8d956819a9.slice/crio-0105b981432502e069eb32798be3a32d9189d9cca32521fc2e432f282b89fa11 WatchSource:0}: Error finding container 0105b981432502e069eb32798be3a32d9189d9cca32521fc2e432f282b89fa11: Status 404 returned error can't find the container with id 0105b981432502e069eb32798be3a32d9189d9cca32521fc2e432f282b89fa11
Sep 12 01:14:10.892077 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:10.892054    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:11.182791 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:11.182760    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm" event={"ID":"39fbd097-cd0a-4b4b-b06c-af8d956819a9","Type":"ContainerStarted","Data":"0105b981432502e069eb32798be3a32d9189d9cca32521fc2e432f282b89fa11"}
Sep 12 01:14:12.189495 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:12.189455    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm" event={"ID":"39fbd097-cd0a-4b4b-b06c-af8d956819a9","Type":"ContainerStarted","Data":"15e338b4a393e31e7cd0a3c9eb2523803d78e5a8f9566a4d17ec4c3c64129213"}
Sep 12 01:14:12.189894 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:12.189525    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:12.208982 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:12.208918    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm" podStartSLOduration=1.832561161 podStartE2EDuration="2.20890418s" totalImagesPullingTime="376.343019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:14:12.207248392 +0000 UTC m=+1856.767700063" watchObservedRunningTime="2026-09-12 01:14:12.20890418 +0000 UTC m=+1856.769355848"
Sep 12 01:14:18.199909 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:18.199880    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:14:21.744526 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:21.744483    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:31.386612 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:31.386577    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:32.706506 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:32.706458    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:32.806237 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:32.806181    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:33.502583 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:33.502554    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:34.405364 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:34.405334    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:35.059849 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:35.059821    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:36.020847 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:36.020812    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:46.216203 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.216121    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"]
Sep 12 01:14:46.219677 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.219657    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.223507 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.223484    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-5fbbcd\"/\"openshift-service-ca.crt\""
Sep 12 01:14:46.223648 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.223486    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-5fbbcd\"/\"ipp-route-5fbbcd-kserve-self-signed-certs\""
Sep 12 01:14:46.223648 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.223536    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-5fbbcd\"/\"kube-root-ca.crt\""
Sep 12 01:14:46.223648 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.223518    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-5fbbcd\"/\"default-dockercfg-pxtsv\""
Sep 12 01:14:46.230229 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.230200    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"]
Sep 12 01:14:46.258508 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.258476    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398acfd7-dea9-4921-9267-a6e3d9485265-tls-certs\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.258669 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.258516    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-tmp-dir\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.258669 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.258603    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-model-cache\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.258755 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.258677    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-home\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.258755 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.258701    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-dshm\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.258755 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.258726    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc6g9\" (UniqueName: \"kubernetes.io/projected/398acfd7-dea9-4921-9267-a6e3d9485265-kube-api-access-dc6g9\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.346582 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.346552    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:14:46.359560 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.359532    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-home\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.359560 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.359563    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-dshm\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.359773 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.359586    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc6g9\" (UniqueName: \"kubernetes.io/projected/398acfd7-dea9-4921-9267-a6e3d9485265-kube-api-access-dc6g9\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.359773 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.359634    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398acfd7-dea9-4921-9267-a6e3d9485265-tls-certs\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.359773 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.359668    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-tmp-dir\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.359773 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.359752    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-model-cache\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.360016 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.359931    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-home\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.360076 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.360054    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-tmp-dir\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.360114 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.360081    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-model-cache\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.361831 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.361802    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-dshm\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.362138 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.362121    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398acfd7-dea9-4921-9267-a6e3d9485265-tls-certs\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.368020 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.367998    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc6g9\" (UniqueName: \"kubernetes.io/projected/398acfd7-dea9-4921-9267-a6e3d9485265-kube-api-access-dc6g9\") pod \"ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") " pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.529028 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.528900    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:46.824940 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:46.824914    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"]
Sep 12 01:14:46.827745 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:14:46.827707    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod398acfd7_dea9_4921_9267_a6e3d9485265.slice/crio-eadd2873cc0328032a1c3f29c515843b3b31d8a5ee7b40c0173b635fde7240f7 WatchSource:0}: Error finding container eadd2873cc0328032a1c3f29c515843b3b31d8a5ee7b40c0173b635fde7240f7: Status 404 returned error can't find the container with id eadd2873cc0328032a1c3f29c515843b3b31d8a5ee7b40c0173b635fde7240f7
Sep 12 01:14:47.356345 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:47.356313    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26" event={"ID":"398acfd7-dea9-4921-9267-a6e3d9485265","Type":"ContainerStarted","Data":"e54ab3b79c9781771c0b4688cd1202a4e8634318a1129434894cf431d8101413"}
Sep 12 01:14:47.356345 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:47.356354    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26" event={"ID":"398acfd7-dea9-4921-9267-a6e3d9485265","Type":"ContainerStarted","Data":"eadd2873cc0328032a1c3f29c515843b3b31d8a5ee7b40c0173b635fde7240f7"}
Sep 12 01:14:47.375914 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:47.375863    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26" podStartSLOduration=1.375843691 podStartE2EDuration="1.375843691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:14:47.374265977 +0000 UTC m=+1891.934717660" watchObservedRunningTime="2026-09-12 01:14:47.375843691 +0000 UTC m=+1891.936295359"
Sep 12 01:14:56.529967 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:56.529918    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:56.530322 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:56.529975    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:56.542365 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:56.542341    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:14:57.415912 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:14:57.415881    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:15:15.538254 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:15.538200    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:15.902687 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:15.902593    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:16.157644 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:16.157612    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:18.611351 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:18.611309    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:19.285409 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:19.285370    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:19.379766 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:19.379731    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:19.480799 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:19.480768    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:20.295104 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:20.295060    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:21.193900 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:21.193868    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:21.989063 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:21.989029    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:22.655799 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.655765    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"]
Sep 12 01:15:22.659365 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.659341    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:22.662192 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.662172    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-bfe85d-serving-cert\""
Sep 12 01:15:22.671208 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.671185    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"]
Sep 12 01:15:22.694728 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.694701    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:22.725249 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.725208    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90b02910-b116-4bfd-9082-35baed5da124-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:22.725471 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.725259    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90b02910-b116-4bfd-9082-35baed5da124-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:22.725471 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.725386    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8df6\" (UniqueName: \"kubernetes.io/projected/90b02910-b116-4bfd-9082-35baed5da124-kube-api-access-s8df6\") pod \"maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:22.725578 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.725471    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90b02910-b116-4bfd-9082-35baed5da124-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:22.794604 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.794572    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:22.826965 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.826906    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8df6\" (UniqueName: \"kubernetes.io/projected/90b02910-b116-4bfd-9082-35baed5da124-kube-api-access-s8df6\") pod \"maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:22.827080 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.827017    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90b02910-b116-4bfd-9082-35baed5da124-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:22.827080 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.827054    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90b02910-b116-4bfd-9082-35baed5da124-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:22.827080 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.827070    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90b02910-b116-4bfd-9082-35baed5da124-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:22.827783 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.827753    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90b02910-b116-4bfd-9082-35baed5da124-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:22.829802 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.829778    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90b02910-b116-4bfd-9082-35baed5da124-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:22.829891 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.829828    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90b02910-b116-4bfd-9082-35baed5da124-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:22.835816 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.835795    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8df6\" (UniqueName: \"kubernetes.io/projected/90b02910-b116-4bfd-9082-35baed5da124-kube-api-access-s8df6\") pod \"maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:22.852939 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.852916    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:22.972813 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:22.972777    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:23.188208 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:23.188175    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:23.253292 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:23.253264    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"]
Sep 12 01:15:23.255217 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:15:23.255190    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b02910_b116_4bfd_9082_35baed5da124.slice/crio-e56d9ac293ca494260086783ba1c1e3ed593dec85f0a8c947f69ff420f057a92 WatchSource:0}: Error finding container e56d9ac293ca494260086783ba1c1e3ed593dec85f0a8c947f69ff420f057a92: Status 404 returned error can't find the container with id e56d9ac293ca494260086783ba1c1e3ed593dec85f0a8c947f69ff420f057a92
Sep 12 01:15:23.263926 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:23.263894    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:23.531500 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:23.531415    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g" event={"ID":"90b02910-b116-4bfd-9082-35baed5da124","Type":"ContainerStarted","Data":"e56d9ac293ca494260086783ba1c1e3ed593dec85f0a8c947f69ff420f057a92"}
Sep 12 01:15:24.537720 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:24.537667    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g" event={"ID":"90b02910-b116-4bfd-9082-35baed5da124","Type":"ContainerStarted","Data":"76268b64822bb4492e957cd8636f4170692110538c8d14b834319e8fe8a2460b"}
Sep 12 01:15:24.538127 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:24.537753    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:24.557544 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:24.557490    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g" podStartSLOduration=2.063788948 podStartE2EDuration="2.557474018s" totalImagesPullingTime="493.68507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:15:24.555003069 +0000 UTC m=+1929.115454753" watchObservedRunningTime="2026-09-12 01:15:24.557474018 +0000 UTC m=+1929.117925685"
Sep 12 01:15:30.548469 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:30.548435    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:15:44.766839 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:44.766803    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:45.610974 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:45.610908    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:46.391184 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:46.391151    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:47.082684 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:47.082628    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:47.917370 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:47.917339    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:48.632911 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:48.632878    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:49.403523 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:49.403489    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:50.950319 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:50.950283    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:51.781410 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:51.781383    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:52.501003 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:52.500964    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:53.317462 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:53.317423    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:54.117931 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:54.117895    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:15:54.808621 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:15:54.808586    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:01.013458 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.013415    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"]
Sep 12 01:16:01.013976 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.013704    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g" podUID="90b02910-b116-4bfd-9082-35baed5da124" containerName="maas-api" containerID="cri-o://76268b64822bb4492e957cd8636f4170692110538c8d14b834319e8fe8a2460b" gracePeriod=30
Sep 12 01:16:01.046589 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.046550    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:01.122148 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.122119    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:01.193541 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.193509    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:01.264311 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.264248    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:16:01.268475 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.268443    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:01.299890 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.299865    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8df6\" (UniqueName: \"kubernetes.io/projected/90b02910-b116-4bfd-9082-35baed5da124-kube-api-access-s8df6\") pod \"90b02910-b116-4bfd-9082-35baed5da124\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") "
Sep 12 01:16:01.300103 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.299923    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90b02910-b116-4bfd-9082-35baed5da124-metrics-certs\") pod \"90b02910-b116-4bfd-9082-35baed5da124\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") "
Sep 12 01:16:01.300103 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.299993    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90b02910-b116-4bfd-9082-35baed5da124-openshift-service-ca\") pod \"90b02910-b116-4bfd-9082-35baed5da124\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") "
Sep 12 01:16:01.300103 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.300045    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90b02910-b116-4bfd-9082-35baed5da124-maas-api-tls\") pod \"90b02910-b116-4bfd-9082-35baed5da124\" (UID: \"90b02910-b116-4bfd-9082-35baed5da124\") "
Sep 12 01:16:01.300434 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.300399    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b02910-b116-4bfd-9082-35baed5da124-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90b02910-b116-4bfd-9082-35baed5da124" (UID: "90b02910-b116-4bfd-9082-35baed5da124"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:16:01.302437 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.302405    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b02910-b116-4bfd-9082-35baed5da124-kube-api-access-s8df6" (OuterVolumeSpecName: "kube-api-access-s8df6") pod "90b02910-b116-4bfd-9082-35baed5da124" (UID: "90b02910-b116-4bfd-9082-35baed5da124"). InnerVolumeSpecName "kube-api-access-s8df6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:16:01.302543 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.302458    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b02910-b116-4bfd-9082-35baed5da124-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90b02910-b116-4bfd-9082-35baed5da124" (UID: "90b02910-b116-4bfd-9082-35baed5da124"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:16:01.302543 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.302509    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b02910-b116-4bfd-9082-35baed5da124-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "90b02910-b116-4bfd-9082-35baed5da124" (UID: "90b02910-b116-4bfd-9082-35baed5da124"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:16:01.328795 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.328767    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:01.401456 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.401421    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90b02910-b116-4bfd-9082-35baed5da124-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:01.401456 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.401453    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90b02910-b116-4bfd-9082-35baed5da124-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:01.401456 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.401462    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8df6\" (UniqueName: \"kubernetes.io/projected/90b02910-b116-4bfd-9082-35baed5da124-kube-api-access-s8df6\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:01.401702 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.401472    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90b02910-b116-4bfd-9082-35baed5da124-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:01.668790 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.668756    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:01.721181 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.721150    2567 generic.go:356] "Generic (PLEG): container finished" podID="90b02910-b116-4bfd-9082-35baed5da124" containerID="76268b64822bb4492e957cd8636f4170692110538c8d14b834319e8fe8a2460b" exitCode=0
Sep 12 01:16:01.721348 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.721194    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"
Sep 12 01:16:01.721348 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.721196    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g" event={"ID":"90b02910-b116-4bfd-9082-35baed5da124","Type":"ContainerDied","Data":"76268b64822bb4492e957cd8636f4170692110538c8d14b834319e8fe8a2460b"}
Sep 12 01:16:01.721348 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.721326    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g" event={"ID":"90b02910-b116-4bfd-9082-35baed5da124","Type":"ContainerDied","Data":"e56d9ac293ca494260086783ba1c1e3ed593dec85f0a8c947f69ff420f057a92"}
Sep 12 01:16:01.721523 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.721353    2567 scope.go:117] "RemoveContainer" containerID="76268b64822bb4492e957cd8636f4170692110538c8d14b834319e8fe8a2460b"
Sep 12 01:16:01.731369 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.731345    2567 scope.go:117] "RemoveContainer" containerID="76268b64822bb4492e957cd8636f4170692110538c8d14b834319e8fe8a2460b"
Sep 12 01:16:01.731693 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:16:01.731670    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76268b64822bb4492e957cd8636f4170692110538c8d14b834319e8fe8a2460b\": container with ID starting with 76268b64822bb4492e957cd8636f4170692110538c8d14b834319e8fe8a2460b not found: ID does not exist" containerID="76268b64822bb4492e957cd8636f4170692110538c8d14b834319e8fe8a2460b"
Sep 12 01:16:01.731767 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.731706    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76268b64822bb4492e957cd8636f4170692110538c8d14b834319e8fe8a2460b"} err="failed to get container status \"76268b64822bb4492e957cd8636f4170692110538c8d14b834319e8fe8a2460b\": rpc error: code = NotFound desc = could not find container \"76268b64822bb4492e957cd8636f4170692110538c8d14b834319e8fe8a2460b\": container with ID starting with 76268b64822bb4492e957cd8636f4170692110538c8d14b834319e8fe8a2460b not found: ID does not exist"
Sep 12 01:16:01.744153 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.744118    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"]
Sep 12 01:16:01.753790 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.753758    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bfe85d-676db5f798-bsh4g"]
Sep 12 01:16:01.782344 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.782312    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:01.974096 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:01.974016    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90b02910-b116-4bfd-9082-35baed5da124" path="/var/lib/kubelet/pods/90b02910-b116-4bfd-9082-35baed5da124/volumes"
Sep 12 01:16:02.601006 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:02.600969    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:03.389586 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:03.389536    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:04.050984 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:04.050934    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:04.724148 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:04.724116    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:07.688164 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:07.688090    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:07.760209 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:07.760181    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:07.950196 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:07.950118    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:11.828313 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:11.828273    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:12.724862 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:12.724829    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:13.593597 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:13.593561    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:14.287665 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:14.287633    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:15.192197 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:15.192154    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:15.892160 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:15.892124    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:27.143124 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.143077    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"]
Sep 12 01:16:27.143702 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.143320    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs" podUID="25fcaa1e-672c-4c6d-8577-ab57dcebdb21" containerName="maas-api" containerID="cri-o://bb20731a1ee65669d1a0f1308e4e2ba73038fc9e3fdff1a95d84d76fbcc95feb" gracePeriod=30
Sep 12 01:16:27.182963 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.182905    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:27.250847 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.250812    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:27.324802 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.324770    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:27.387373 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.387342    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:16:27.392671 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.392644    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:27.450853 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.450825    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:27.456905 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.456884    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54jbt\" (UniqueName: \"kubernetes.io/projected/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-kube-api-access-54jbt\") pod \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") "
Sep 12 01:16:27.457031 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.456926    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-openshift-service-ca\") pod \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") "
Sep 12 01:16:27.457079 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.457040    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-metrics-certs\") pod \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") "
Sep 12 01:16:27.457079 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.457061    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-maas-api-tls\") pod \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\" (UID: \"25fcaa1e-672c-4c6d-8577-ab57dcebdb21\") "
Sep 12 01:16:27.457375 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.457340    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25fcaa1e-672c-4c6d-8577-ab57dcebdb21" (UID: "25fcaa1e-672c-4c6d-8577-ab57dcebdb21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:16:27.459282 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.459257    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "25fcaa1e-672c-4c6d-8577-ab57dcebdb21" (UID: "25fcaa1e-672c-4c6d-8577-ab57dcebdb21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:16:27.459282 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.459272    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25fcaa1e-672c-4c6d-8577-ab57dcebdb21" (UID: "25fcaa1e-672c-4c6d-8577-ab57dcebdb21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:16:27.459404 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.459329    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-kube-api-access-54jbt" (OuterVolumeSpecName: "kube-api-access-54jbt") pod "25fcaa1e-672c-4c6d-8577-ab57dcebdb21" (UID: "25fcaa1e-672c-4c6d-8577-ab57dcebdb21"). InnerVolumeSpecName "kube-api-access-54jbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:16:27.558195 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.558164    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54jbt\" (UniqueName: \"kubernetes.io/projected/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-kube-api-access-54jbt\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:27.558195 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.558188    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:27.558195 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.558198    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:27.558195 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.558206    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fcaa1e-672c-4c6d-8577-ab57dcebdb21-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:27.801241 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.801152    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:27.846255 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.846221    2567 generic.go:356] "Generic (PLEG): container finished" podID="25fcaa1e-672c-4c6d-8577-ab57dcebdb21" containerID="bb20731a1ee65669d1a0f1308e4e2ba73038fc9e3fdff1a95d84d76fbcc95feb" exitCode=0
Sep 12 01:16:27.846443 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.846257    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs" event={"ID":"25fcaa1e-672c-4c6d-8577-ab57dcebdb21","Type":"ContainerDied","Data":"bb20731a1ee65669d1a0f1308e4e2ba73038fc9e3fdff1a95d84d76fbcc95feb"}
Sep 12 01:16:27.846443 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.846303    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"
Sep 12 01:16:27.846443 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.846322    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs" event={"ID":"25fcaa1e-672c-4c6d-8577-ab57dcebdb21","Type":"ContainerDied","Data":"6a689ac5cfc97788f1694558dfe1bbea07854ca8fcd8171a6aa409deed19644b"}
Sep 12 01:16:27.846443 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.846340    2567 scope.go:117] "RemoveContainer" containerID="bb20731a1ee65669d1a0f1308e4e2ba73038fc9e3fdff1a95d84d76fbcc95feb"
Sep 12 01:16:27.856370 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.856350    2567 scope.go:117] "RemoveContainer" containerID="bb20731a1ee65669d1a0f1308e4e2ba73038fc9e3fdff1a95d84d76fbcc95feb"
Sep 12 01:16:27.856676 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:16:27.856651    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb20731a1ee65669d1a0f1308e4e2ba73038fc9e3fdff1a95d84d76fbcc95feb\": container with ID starting with bb20731a1ee65669d1a0f1308e4e2ba73038fc9e3fdff1a95d84d76fbcc95feb not found: ID does not exist" containerID="bb20731a1ee65669d1a0f1308e4e2ba73038fc9e3fdff1a95d84d76fbcc95feb"
Sep 12 01:16:27.856676 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.856682    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb20731a1ee65669d1a0f1308e4e2ba73038fc9e3fdff1a95d84d76fbcc95feb"} err="failed to get container status \"bb20731a1ee65669d1a0f1308e4e2ba73038fc9e3fdff1a95d84d76fbcc95feb\": rpc error: code = NotFound desc = could not find container \"bb20731a1ee65669d1a0f1308e4e2ba73038fc9e3fdff1a95d84d76fbcc95feb\": container with ID starting with bb20731a1ee65669d1a0f1308e4e2ba73038fc9e3fdff1a95d84d76fbcc95feb not found: ID does not exist"
Sep 12 01:16:27.870471 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.870447    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"]
Sep 12 01:16:27.875427 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.875405    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbbcd-8457856c7c-qhsgs"]
Sep 12 01:16:27.882185 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.882161    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:27.974192 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:27.974159    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25fcaa1e-672c-4c6d-8577-ab57dcebdb21" path="/var/lib/kubelet/pods/25fcaa1e-672c-4c6d-8577-ab57dcebdb21/volumes"
Sep 12 01:16:28.489417 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:28.489381    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:29.191930 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:29.191895    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:29.659850 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:29.659818    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:30.000346 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:30.000261    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:32.759491 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:32.759459    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"]
Sep 12 01:16:32.855018 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:32.854985    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:16:47.779435 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:47.779386    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26" podUID="398acfd7-dea9-4921-9267-a6e3d9485265" containerName="main" containerID="cri-o://e54ab3b79c9781771c0b4688cd1202a4e8634318a1129434894cf431d8101413" gracePeriod=45
Sep 12 01:16:47.942323 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:47.942283    2567 generic.go:356] "Generic (PLEG): container finished" podID="398acfd7-dea9-4921-9267-a6e3d9485265" containerID="e54ab3b79c9781771c0b4688cd1202a4e8634318a1129434894cf431d8101413" exitCode=0
Sep 12 01:16:47.942506 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:47.942350    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26" event={"ID":"398acfd7-dea9-4921-9267-a6e3d9485265","Type":"ContainerDied","Data":"e54ab3b79c9781771c0b4688cd1202a4e8634318a1129434894cf431d8101413"}
Sep 12 01:16:48.044211 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.044141    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:16:48.152487 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.152449    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dc6g9\" (UniqueName: \"kubernetes.io/projected/398acfd7-dea9-4921-9267-a6e3d9485265-kube-api-access-dc6g9\") pod \"398acfd7-dea9-4921-9267-a6e3d9485265\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") "
Sep 12 01:16:48.152487 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.152486    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-model-cache\") pod \"398acfd7-dea9-4921-9267-a6e3d9485265\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") "
Sep 12 01:16:48.152719 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.152510    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-tmp-dir\") pod \"398acfd7-dea9-4921-9267-a6e3d9485265\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") "
Sep 12 01:16:48.152719 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.152615    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398acfd7-dea9-4921-9267-a6e3d9485265-tls-certs\") pod \"398acfd7-dea9-4921-9267-a6e3d9485265\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") "
Sep 12 01:16:48.152719 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.152692    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-home\") pod \"398acfd7-dea9-4921-9267-a6e3d9485265\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") "
Sep 12 01:16:48.152719 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.152719    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-dshm\") pod \"398acfd7-dea9-4921-9267-a6e3d9485265\" (UID: \"398acfd7-dea9-4921-9267-a6e3d9485265\") "
Sep 12 01:16:48.152942 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.152762    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "398acfd7-dea9-4921-9267-a6e3d9485265" (UID: "398acfd7-dea9-4921-9267-a6e3d9485265"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:16:48.152942 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.152780    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-model-cache" (OuterVolumeSpecName: "model-cache") pod "398acfd7-dea9-4921-9267-a6e3d9485265" (UID: "398acfd7-dea9-4921-9267-a6e3d9485265"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:16:48.153083 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.152995    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-home" (OuterVolumeSpecName: "home") pod "398acfd7-dea9-4921-9267-a6e3d9485265" (UID: "398acfd7-dea9-4921-9267-a6e3d9485265"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:16:48.153199 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.153180    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-model-cache\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:48.153259 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.153203    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-tmp-dir\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:48.153259 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.153211    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-home\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:48.154912 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.154889    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/398acfd7-dea9-4921-9267-a6e3d9485265-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "398acfd7-dea9-4921-9267-a6e3d9485265" (UID: "398acfd7-dea9-4921-9267-a6e3d9485265"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:16:48.155066 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.154926    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-dshm" (OuterVolumeSpecName: "dshm") pod "398acfd7-dea9-4921-9267-a6e3d9485265" (UID: "398acfd7-dea9-4921-9267-a6e3d9485265"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:16:48.155066 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.155003    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398acfd7-dea9-4921-9267-a6e3d9485265-kube-api-access-dc6g9" (OuterVolumeSpecName: "kube-api-access-dc6g9") pod "398acfd7-dea9-4921-9267-a6e3d9485265" (UID: "398acfd7-dea9-4921-9267-a6e3d9485265"). InnerVolumeSpecName "kube-api-access-dc6g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:16:48.254138 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.254104    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398acfd7-dea9-4921-9267-a6e3d9485265-dshm\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:48.254138 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.254134    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc6g9\" (UniqueName: \"kubernetes.io/projected/398acfd7-dea9-4921-9267-a6e3d9485265-kube-api-access-dc6g9\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:48.254138 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.254145    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398acfd7-dea9-4921-9267-a6e3d9485265-tls-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:48.948070 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.948039    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"
Sep 12 01:16:48.948070 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.948047    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26" event={"ID":"398acfd7-dea9-4921-9267-a6e3d9485265","Type":"ContainerDied","Data":"eadd2873cc0328032a1c3f29c515843b3b31d8a5ee7b40c0173b635fde7240f7"}
Sep 12 01:16:48.948661 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.948094    2567 scope.go:117] "RemoveContainer" containerID="e54ab3b79c9781771c0b4688cd1202a4e8634318a1129434894cf431d8101413"
Sep 12 01:16:48.970723 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.970699    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"]
Sep 12 01:16:48.974197 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:48.974173    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-5fbbcd/ipp-route-5fbbcd-kserve-7676dbdbdc-2wz26"]
Sep 12 01:16:49.973615 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:16:49.973577    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="398acfd7-dea9-4921-9267-a6e3d9485265" path="/var/lib/kubelet/pods/398acfd7-dea9-4921-9267-a6e3d9485265/volumes"
Sep 12 01:17:08.901570 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:08.901531    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:10.152850 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:10.152812    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:15.751255 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:15.751184    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:15.852407 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:15.852379    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:16.042091 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:16.041980    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:16.740659 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:16.740621    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:17.582869 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:17.582818    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:18.315601 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:18.315551    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:19.023959 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:19.023920    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:19.823151 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:19.823106    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:26.863390 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:26.863352    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"]
Sep 12 01:17:26.864276 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:26.864243    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm" podUID="39fbd097-cd0a-4b4b-b06c-af8d956819a9" containerName="maas-api" containerID="cri-o://15e338b4a393e31e7cd0a3c9eb2523803d78e5a8f9566a4d17ec4c3c64129213" gracePeriod=30
Sep 12 01:17:26.895126 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:26.895098    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:26.964628 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:26.964592    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:27.062233 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.062196    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:27.115688 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.115626    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:17:27.123696 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.123668    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:27.128904 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.128881    2567 generic.go:356] "Generic (PLEG): container finished" podID="39fbd097-cd0a-4b4b-b06c-af8d956819a9" containerID="15e338b4a393e31e7cd0a3c9eb2523803d78e5a8f9566a4d17ec4c3c64129213" exitCode=0
Sep 12 01:17:27.129045 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.128913    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"
Sep 12 01:17:27.129045 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.128973    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm" event={"ID":"39fbd097-cd0a-4b4b-b06c-af8d956819a9","Type":"ContainerDied","Data":"15e338b4a393e31e7cd0a3c9eb2523803d78e5a8f9566a4d17ec4c3c64129213"}
Sep 12 01:17:27.129045 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.129020    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm" event={"ID":"39fbd097-cd0a-4b4b-b06c-af8d956819a9","Type":"ContainerDied","Data":"0105b981432502e069eb32798be3a32d9189d9cca32521fc2e432f282b89fa11"}
Sep 12 01:17:27.129045 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.129043    2567 scope.go:117] "RemoveContainer" containerID="15e338b4a393e31e7cd0a3c9eb2523803d78e5a8f9566a4d17ec4c3c64129213"
Sep 12 01:17:27.139168 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.139150    2567 scope.go:117] "RemoveContainer" containerID="15e338b4a393e31e7cd0a3c9eb2523803d78e5a8f9566a4d17ec4c3c64129213"
Sep 12 01:17:27.139495 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:17:27.139477    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e338b4a393e31e7cd0a3c9eb2523803d78e5a8f9566a4d17ec4c3c64129213\": container with ID starting with 15e338b4a393e31e7cd0a3c9eb2523803d78e5a8f9566a4d17ec4c3c64129213 not found: ID does not exist" containerID="15e338b4a393e31e7cd0a3c9eb2523803d78e5a8f9566a4d17ec4c3c64129213"
Sep 12 01:17:27.139571 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.139502    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e338b4a393e31e7cd0a3c9eb2523803d78e5a8f9566a4d17ec4c3c64129213"} err="failed to get container status \"15e338b4a393e31e7cd0a3c9eb2523803d78e5a8f9566a4d17ec4c3c64129213\": rpc error: code = NotFound desc = could not find container \"15e338b4a393e31e7cd0a3c9eb2523803d78e5a8f9566a4d17ec4c3c64129213\": container with ID starting with 15e338b4a393e31e7cd0a3c9eb2523803d78e5a8f9566a4d17ec4c3c64129213 not found: ID does not exist"
Sep 12 01:17:27.234767 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.234728    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39fbd097-cd0a-4b4b-b06c-af8d956819a9-metrics-certs\") pod \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") "
Sep 12 01:17:27.234976 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.234801    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8pt5\" (UniqueName: \"kubernetes.io/projected/39fbd097-cd0a-4b4b-b06c-af8d956819a9-kube-api-access-h8pt5\") pod \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") "
Sep 12 01:17:27.234976 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.234826    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39fbd097-cd0a-4b4b-b06c-af8d956819a9-maas-api-tls\") pod \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") "
Sep 12 01:17:27.234976 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.234851    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fbd097-cd0a-4b4b-b06c-af8d956819a9-openshift-service-ca\") pod \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\" (UID: \"39fbd097-cd0a-4b4b-b06c-af8d956819a9\") "
Sep 12 01:17:27.235341 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.235314    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39fbd097-cd0a-4b4b-b06c-af8d956819a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "39fbd097-cd0a-4b4b-b06c-af8d956819a9" (UID: "39fbd097-cd0a-4b4b-b06c-af8d956819a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:17:27.237164 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.237139    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39fbd097-cd0a-4b4b-b06c-af8d956819a9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "39fbd097-cd0a-4b4b-b06c-af8d956819a9" (UID: "39fbd097-cd0a-4b4b-b06c-af8d956819a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:17:27.237478 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.237444    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39fbd097-cd0a-4b4b-b06c-af8d956819a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "39fbd097-cd0a-4b4b-b06c-af8d956819a9" (UID: "39fbd097-cd0a-4b4b-b06c-af8d956819a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:17:27.238330 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.238314    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39fbd097-cd0a-4b4b-b06c-af8d956819a9-kube-api-access-h8pt5" (OuterVolumeSpecName: "kube-api-access-h8pt5") pod "39fbd097-cd0a-4b4b-b06c-af8d956819a9" (UID: "39fbd097-cd0a-4b4b-b06c-af8d956819a9"). InnerVolumeSpecName "kube-api-access-h8pt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:17:27.336269 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.336236    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8pt5\" (UniqueName: \"kubernetes.io/projected/39fbd097-cd0a-4b4b-b06c-af8d956819a9-kube-api-access-h8pt5\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:27.336269 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.336268    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39fbd097-cd0a-4b4b-b06c-af8d956819a9-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:27.336269 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.336277    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fbd097-cd0a-4b4b-b06c-af8d956819a9-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:27.336519 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.336286    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39fbd097-cd0a-4b4b-b06c-af8d956819a9-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:27.454254 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.454221    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"]
Sep 12 01:17:27.458988 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.458962    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-57b051-7c5f67d78d-fjqqm"]
Sep 12 01:17:27.492741 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.492708    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:27.593353 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.593320    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:27.974378 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:27.974349    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39fbd097-cd0a-4b4b-b06c-af8d956819a9" path="/var/lib/kubelet/pods/39fbd097-cd0a-4b4b-b06c-af8d956819a9/volumes"
Sep 12 01:17:28.452356 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:28.452316    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:29.187034 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:29.187005    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:33.384476 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:33.384437    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:33.484675 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:33.484647    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:33.649767 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:33.649679    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:34.300423 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:34.300391    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:35.110207 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:35.110159    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:35.843832 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:35.843799    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:36.685201 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:36.685122    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:41.625893 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:41.625856    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:44.575224 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.575187    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"]
Sep 12 01:17:44.575624 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.575397    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6" podUID="707d2410-20a8-49b8-ac59-553f5c12341d" containerName="maas-api" containerID="cri-o://60200cd203815ea51828e8ee24fc3c9d5de93f9fa71f67f2f5f5642d7b054e87" gracePeriod=30
Sep 12 01:17:44.610066 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.610038    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:44.708161 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.708125    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:44.810316 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.810287    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:44.823552 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.823530    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:17:44.894570 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.894469    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/707d2410-20a8-49b8-ac59-553f5c12341d-openshift-service-ca\") pod \"707d2410-20a8-49b8-ac59-553f5c12341d\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") "
Sep 12 01:17:44.894570 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.894507    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-maas-api-tls\") pod \"707d2410-20a8-49b8-ac59-553f5c12341d\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") "
Sep 12 01:17:44.894570 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.894546    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdf2w\" (UniqueName: \"kubernetes.io/projected/707d2410-20a8-49b8-ac59-553f5c12341d-kube-api-access-mdf2w\") pod \"707d2410-20a8-49b8-ac59-553f5c12341d\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") "
Sep 12 01:17:44.894841 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.894589    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-metrics-certs\") pod \"707d2410-20a8-49b8-ac59-553f5c12341d\" (UID: \"707d2410-20a8-49b8-ac59-553f5c12341d\") "
Sep 12 01:17:44.894919 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.894885    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707d2410-20a8-49b8-ac59-553f5c12341d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "707d2410-20a8-49b8-ac59-553f5c12341d" (UID: "707d2410-20a8-49b8-ac59-553f5c12341d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:17:44.896795 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.896765    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "707d2410-20a8-49b8-ac59-553f5c12341d" (UID: "707d2410-20a8-49b8-ac59-553f5c12341d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:17:44.896892 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.896804    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "707d2410-20a8-49b8-ac59-553f5c12341d" (UID: "707d2410-20a8-49b8-ac59-553f5c12341d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:17:44.896892 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.896853    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707d2410-20a8-49b8-ac59-553f5c12341d-kube-api-access-mdf2w" (OuterVolumeSpecName: "kube-api-access-mdf2w") pod "707d2410-20a8-49b8-ac59-553f5c12341d" (UID: "707d2410-20a8-49b8-ac59-553f5c12341d"). InnerVolumeSpecName "kube-api-access-mdf2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:17:44.908117 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.908094    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:44.995306 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.995267    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:44.995306 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.995300    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/707d2410-20a8-49b8-ac59-553f5c12341d-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:44.995306 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.995311    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/707d2410-20a8-49b8-ac59-553f5c12341d-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:44.995547 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:44.995321    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdf2w\" (UniqueName: \"kubernetes.io/projected/707d2410-20a8-49b8-ac59-553f5c12341d-kube-api-access-mdf2w\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:45.217431 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:45.217397    2567 generic.go:356] "Generic (PLEG): container finished" podID="707d2410-20a8-49b8-ac59-553f5c12341d" containerID="60200cd203815ea51828e8ee24fc3c9d5de93f9fa71f67f2f5f5642d7b054e87" exitCode=0
Sep 12 01:17:45.217603 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:45.217463    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"
Sep 12 01:17:45.217603 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:45.217471    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6" event={"ID":"707d2410-20a8-49b8-ac59-553f5c12341d","Type":"ContainerDied","Data":"60200cd203815ea51828e8ee24fc3c9d5de93f9fa71f67f2f5f5642d7b054e87"}
Sep 12 01:17:45.217603 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:45.217513    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6" event={"ID":"707d2410-20a8-49b8-ac59-553f5c12341d","Type":"ContainerDied","Data":"5a4a37d56689fc4c4ba2fb981667897cf25f78b10d19113468851f4f543d44f5"}
Sep 12 01:17:45.217603 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:45.217533    2567 scope.go:117] "RemoveContainer" containerID="60200cd203815ea51828e8ee24fc3c9d5de93f9fa71f67f2f5f5642d7b054e87"
Sep 12 01:17:45.226986 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:45.226942    2567 scope.go:117] "RemoveContainer" containerID="60200cd203815ea51828e8ee24fc3c9d5de93f9fa71f67f2f5f5642d7b054e87"
Sep 12 01:17:45.227250 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:17:45.227226    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60200cd203815ea51828e8ee24fc3c9d5de93f9fa71f67f2f5f5642d7b054e87\": container with ID starting with 60200cd203815ea51828e8ee24fc3c9d5de93f9fa71f67f2f5f5642d7b054e87 not found: ID does not exist" containerID="60200cd203815ea51828e8ee24fc3c9d5de93f9fa71f67f2f5f5642d7b054e87"
Sep 12 01:17:45.227328 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:45.227259    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60200cd203815ea51828e8ee24fc3c9d5de93f9fa71f67f2f5f5642d7b054e87"} err="failed to get container status \"60200cd203815ea51828e8ee24fc3c9d5de93f9fa71f67f2f5f5642d7b054e87\": rpc error: code = NotFound desc = could not find container \"60200cd203815ea51828e8ee24fc3c9d5de93f9fa71f67f2f5f5642d7b054e87\": container with ID starting with 60200cd203815ea51828e8ee24fc3c9d5de93f9fa71f67f2f5f5642d7b054e87 not found: ID does not exist"
Sep 12 01:17:45.230401 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:45.230384    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:45.242133 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:45.242107    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"]
Sep 12 01:17:45.251328 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:45.251310    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-61f73f-675cd7fb97-jh9z6"]
Sep 12 01:17:45.336472 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:45.336438    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:45.974295 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:45.974265    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="707d2410-20a8-49b8-ac59-553f5c12341d" path="/var/lib/kubelet/pods/707d2410-20a8-49b8-ac59-553f5c12341d/volumes"
Sep 12 01:17:46.088712 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:46.088680    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:51.502448 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:51.502410    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:51.601197 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:51.601155    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:51.795425 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:51.795337    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:52.401780 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:52.401750    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:53.247155 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:53.247119    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:17:53.910991 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:17:53.910935    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:01.291302 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:01.291255    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:01.349238 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:01.349208    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:02.606403 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.606366    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"]
Sep 12 01:18:02.606884 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.606630    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n" podUID="c8f888b9-c82d-413f-8c22-0605cff89933" containerName="maas-api" containerID="cri-o://ff1b474aa7e782bedb4b54b2c420cd0640831f7543174e5d3a8a30cf65dd47c6" gracePeriod=30
Sep 12 01:18:02.633424 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.633396    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:02.707773 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.707726    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:02.812476 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.812449    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:02.850010 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.849983    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:18:02.875181 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.874907    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:02.954895 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.954864    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-maas-api-tls\") pod \"c8f888b9-c82d-413f-8c22-0605cff89933\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") "
Sep 12 01:18:02.955115 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.954943    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bnpg\" (UniqueName: \"kubernetes.io/projected/c8f888b9-c82d-413f-8c22-0605cff89933-kube-api-access-7bnpg\") pod \"c8f888b9-c82d-413f-8c22-0605cff89933\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") "
Sep 12 01:18:02.955115 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.955002    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-metrics-certs\") pod \"c8f888b9-c82d-413f-8c22-0605cff89933\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") "
Sep 12 01:18:02.955115 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.955044    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f888b9-c82d-413f-8c22-0605cff89933-openshift-service-ca\") pod \"c8f888b9-c82d-413f-8c22-0605cff89933\" (UID: \"c8f888b9-c82d-413f-8c22-0605cff89933\") "
Sep 12 01:18:02.955464 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.955440    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f888b9-c82d-413f-8c22-0605cff89933-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8f888b9-c82d-413f-8c22-0605cff89933" (UID: "c8f888b9-c82d-413f-8c22-0605cff89933"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:18:02.957368 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.957342    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c8f888b9-c82d-413f-8c22-0605cff89933" (UID: "c8f888b9-c82d-413f-8c22-0605cff89933"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:18:02.957848 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.957818    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f888b9-c82d-413f-8c22-0605cff89933-kube-api-access-7bnpg" (OuterVolumeSpecName: "kube-api-access-7bnpg") pod "c8f888b9-c82d-413f-8c22-0605cff89933" (UID: "c8f888b9-c82d-413f-8c22-0605cff89933"). InnerVolumeSpecName "kube-api-access-7bnpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:18:02.957848 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:02.957833    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8f888b9-c82d-413f-8c22-0605cff89933" (UID: "c8f888b9-c82d-413f-8c22-0605cff89933"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:18:03.056385 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.056337    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:18:03.056385 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.056380    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bnpg\" (UniqueName: \"kubernetes.io/projected/c8f888b9-c82d-413f-8c22-0605cff89933-kube-api-access-7bnpg\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:18:03.056385 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.056390    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f888b9-c82d-413f-8c22-0605cff89933-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:18:03.056636 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.056486    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f888b9-c82d-413f-8c22-0605cff89933-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:18:03.249002 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.248943    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:03.305851 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.305814    2567 generic.go:356] "Generic (PLEG): container finished" podID="c8f888b9-c82d-413f-8c22-0605cff89933" containerID="ff1b474aa7e782bedb4b54b2c420cd0640831f7543174e5d3a8a30cf65dd47c6" exitCode=0
Sep 12 01:18:03.306055 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.305860    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"
Sep 12 01:18:03.306055 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.305892    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n" event={"ID":"c8f888b9-c82d-413f-8c22-0605cff89933","Type":"ContainerDied","Data":"ff1b474aa7e782bedb4b54b2c420cd0640831f7543174e5d3a8a30cf65dd47c6"}
Sep 12 01:18:03.306055 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.305968    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n" event={"ID":"c8f888b9-c82d-413f-8c22-0605cff89933","Type":"ContainerDied","Data":"57b414e4ad927875cc5a2d4df42037f0f85fac4c60072dfefc979530061062f9"}
Sep 12 01:18:03.306055 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.305975    2567 scope.go:117] "RemoveContainer" containerID="ff1b474aa7e782bedb4b54b2c420cd0640831f7543174e5d3a8a30cf65dd47c6"
Sep 12 01:18:03.315694 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.315677    2567 scope.go:117] "RemoveContainer" containerID="ff1b474aa7e782bedb4b54b2c420cd0640831f7543174e5d3a8a30cf65dd47c6"
Sep 12 01:18:03.316010 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:18:03.315990    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1b474aa7e782bedb4b54b2c420cd0640831f7543174e5d3a8a30cf65dd47c6\": container with ID starting with ff1b474aa7e782bedb4b54b2c420cd0640831f7543174e5d3a8a30cf65dd47c6 not found: ID does not exist" containerID="ff1b474aa7e782bedb4b54b2c420cd0640831f7543174e5d3a8a30cf65dd47c6"
Sep 12 01:18:03.316080 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.316018    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1b474aa7e782bedb4b54b2c420cd0640831f7543174e5d3a8a30cf65dd47c6"} err="failed to get container status \"ff1b474aa7e782bedb4b54b2c420cd0640831f7543174e5d3a8a30cf65dd47c6\": rpc error: code = NotFound desc = could not find container \"ff1b474aa7e782bedb4b54b2c420cd0640831f7543174e5d3a8a30cf65dd47c6\": container with ID starting with ff1b474aa7e782bedb4b54b2c420cd0640831f7543174e5d3a8a30cf65dd47c6 not found: ID does not exist"
Sep 12 01:18:03.327656 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.327625    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"]
Sep 12 01:18:03.333149 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.333121    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:03.333289 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.333271    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5997-c5b466664-2fq6n"]
Sep 12 01:18:03.973746 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:03.973714    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8f888b9-c82d-413f-8c22-0605cff89933" path="/var/lib/kubelet/pods/c8f888b9-c82d-413f-8c22-0605cff89933/volumes"
Sep 12 01:18:09.074415 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:09.074384    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:09.175025 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:09.174986    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:09.342670 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:09.342561    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:16.043101 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:16.043075    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 01:18:16.045605 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:16.045580    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 01:18:28.537892 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:28.537857    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:28.597211 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:28.597172    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:43.252462 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:43.252408    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:18:50.544627 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:18:50.544594    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:19:33.340701 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:19:33.340584    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:19:34.994120 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:19:34.994091    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:19:51.103886 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:19:51.103851    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:22:24.599013 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:22:24.598901    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:22:42.372030 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:22:42.371997    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:23:10.748893 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:10.748854    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:23:12.264116 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:12.264085    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:23:16.072094 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:16.072064    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 01:23:16.078240 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:16.078214    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 01:23:29.394452 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394416    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"]
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394768    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fcaa1e-672c-4c6d-8577-ab57dcebdb21" containerName="maas-api"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394779    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fcaa1e-672c-4c6d-8577-ab57dcebdb21" containerName="maas-api"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394794    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b02910-b116-4bfd-9082-35baed5da124" containerName="maas-api"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394799    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b02910-b116-4bfd-9082-35baed5da124" containerName="maas-api"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394809    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8f888b9-c82d-413f-8c22-0605cff89933" containerName="maas-api"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394816    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8f888b9-c82d-413f-8c22-0605cff89933" containerName="maas-api"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394823    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="707d2410-20a8-49b8-ac59-553f5c12341d" containerName="maas-api"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394828    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="707d2410-20a8-49b8-ac59-553f5c12341d" containerName="maas-api"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394839    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="398acfd7-dea9-4921-9267-a6e3d9485265" containerName="main"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394845    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="398acfd7-dea9-4921-9267-a6e3d9485265" containerName="main"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394851    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39fbd097-cd0a-4b4b-b06c-af8d956819a9" containerName="maas-api"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394856    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="39fbd097-cd0a-4b4b-b06c-af8d956819a9" containerName="maas-api"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394913    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8f888b9-c82d-413f-8c22-0605cff89933" containerName="maas-api"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394924    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="398acfd7-dea9-4921-9267-a6e3d9485265" containerName="main"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394930    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="39fbd097-cd0a-4b4b-b06c-af8d956819a9" containerName="maas-api"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394935    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="707d2410-20a8-49b8-ac59-553f5c12341d" containerName="maas-api"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394941    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fcaa1e-672c-4c6d-8577-ab57dcebdb21" containerName="maas-api"
Sep 12 01:23:29.396821 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.394970    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b02910-b116-4bfd-9082-35baed5da124" containerName="maas-api"
Sep 12 01:23:29.397841 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.397824    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:29.409532 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.409507    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"]
Sep 12 01:23:29.502410 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.502373    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c1efb3-b874-421b-801b-b6ff16a67180-metrics-certs\") pod \"maas-api-749bf69647-mnx6k\" (UID: \"31c1efb3-b874-421b-801b-b6ff16a67180\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:29.502586 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.502423    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbv5h\" (UniqueName: \"kubernetes.io/projected/31c1efb3-b874-421b-801b-b6ff16a67180-kube-api-access-nbv5h\") pod \"maas-api-749bf69647-mnx6k\" (UID: \"31c1efb3-b874-421b-801b-b6ff16a67180\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:29.502586 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.502458    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31c1efb3-b874-421b-801b-b6ff16a67180-maas-api-tls\") pod \"maas-api-749bf69647-mnx6k\" (UID: \"31c1efb3-b874-421b-801b-b6ff16a67180\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:29.502586 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.502533    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c1efb3-b874-421b-801b-b6ff16a67180-openshift-service-ca\") pod \"maas-api-749bf69647-mnx6k\" (UID: \"31c1efb3-b874-421b-801b-b6ff16a67180\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:29.603775 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.603739    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c1efb3-b874-421b-801b-b6ff16a67180-openshift-service-ca\") pod \"maas-api-749bf69647-mnx6k\" (UID: \"31c1efb3-b874-421b-801b-b6ff16a67180\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:29.603924 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.603799    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c1efb3-b874-421b-801b-b6ff16a67180-metrics-certs\") pod \"maas-api-749bf69647-mnx6k\" (UID: \"31c1efb3-b874-421b-801b-b6ff16a67180\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:29.603924 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.603833    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbv5h\" (UniqueName: \"kubernetes.io/projected/31c1efb3-b874-421b-801b-b6ff16a67180-kube-api-access-nbv5h\") pod \"maas-api-749bf69647-mnx6k\" (UID: \"31c1efb3-b874-421b-801b-b6ff16a67180\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:29.603924 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.603869    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31c1efb3-b874-421b-801b-b6ff16a67180-maas-api-tls\") pod \"maas-api-749bf69647-mnx6k\" (UID: \"31c1efb3-b874-421b-801b-b6ff16a67180\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:29.604403 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.604374    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c1efb3-b874-421b-801b-b6ff16a67180-openshift-service-ca\") pod \"maas-api-749bf69647-mnx6k\" (UID: \"31c1efb3-b874-421b-801b-b6ff16a67180\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:29.606419 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.606398    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31c1efb3-b874-421b-801b-b6ff16a67180-maas-api-tls\") pod \"maas-api-749bf69647-mnx6k\" (UID: \"31c1efb3-b874-421b-801b-b6ff16a67180\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:29.606520 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.606462    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c1efb3-b874-421b-801b-b6ff16a67180-metrics-certs\") pod \"maas-api-749bf69647-mnx6k\" (UID: \"31c1efb3-b874-421b-801b-b6ff16a67180\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:29.613902 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.613876    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbv5h\" (UniqueName: \"kubernetes.io/projected/31c1efb3-b874-421b-801b-b6ff16a67180-kube-api-access-nbv5h\") pod \"maas-api-749bf69647-mnx6k\" (UID: \"31c1efb3-b874-421b-801b-b6ff16a67180\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:29.709063 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.709025    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:29.984188 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.984156    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"]
Sep 12 01:23:29.986443 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:23:29.986406    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c1efb3_b874_421b_801b_b6ff16a67180.slice/crio-4bc2d4203d9dd98143c2e8d369cd8b6c5ab3efafbc3dc46dbb66f52c775c5332 WatchSource:0}: Error finding container 4bc2d4203d9dd98143c2e8d369cd8b6c5ab3efafbc3dc46dbb66f52c775c5332: Status 404 returned error can't find the container with id 4bc2d4203d9dd98143c2e8d369cd8b6c5ab3efafbc3dc46dbb66f52c775c5332
Sep 12 01:23:29.987854 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:29.987837    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 01:23:30.814210 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:30.814176    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k" event={"ID":"31c1efb3-b874-421b-801b-b6ff16a67180","Type":"ContainerStarted","Data":"d9a73b05cd49a4f7b43032af0e3224027c3baec34162a11c97b9f37748f43cbd"}
Sep 12 01:23:30.814602 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:30.814227    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k" event={"ID":"31c1efb3-b874-421b-801b-b6ff16a67180","Type":"ContainerStarted","Data":"4bc2d4203d9dd98143c2e8d369cd8b6c5ab3efafbc3dc46dbb66f52c775c5332"}
Sep 12 01:23:30.814602 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:30.814258    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:30.831471 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:30.831419    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k" podStartSLOduration=1.428714837 podStartE2EDuration="1.831403727s" totalImagesPullingTime="402.68889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:23:30.829625457 +0000 UTC m=+2415.390077146" watchObservedRunningTime="2026-09-12 01:23:30.831403727 +0000 UTC m=+2415.391855394"
Sep 12 01:23:36.824175 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:36.824103    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-749bf69647-mnx6k"
Sep 12 01:23:44.425760 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:44.425717    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"]
Sep 12 01:23:44.426169 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:44.426025    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl" podUID="483879f7-8946-4dc9-950f-64f8082d9651" containerName="maas-api" containerID="cri-o://e6496b37c642744a553bd3953cdcb0d5c11135d4a87705d4fd2350a2a745b3f4" gracePeriod=30
Sep 12 01:23:45.671996 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.671937    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 01:23:45.754827 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.754799    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483879f7-8946-4dc9-950f-64f8082d9651-openshift-service-ca\") pod \"483879f7-8946-4dc9-950f-64f8082d9651\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") "
Sep 12 01:23:45.755020 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.754878    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483879f7-8946-4dc9-950f-64f8082d9651-maas-api-tls\") pod \"483879f7-8946-4dc9-950f-64f8082d9651\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") "
Sep 12 01:23:45.755020 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.754994    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483879f7-8946-4dc9-950f-64f8082d9651-metrics-certs\") pod \"483879f7-8946-4dc9-950f-64f8082d9651\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") "
Sep 12 01:23:45.755115 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.755072    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6sll\" (UniqueName: \"kubernetes.io/projected/483879f7-8946-4dc9-950f-64f8082d9651-kube-api-access-h6sll\") pod \"483879f7-8946-4dc9-950f-64f8082d9651\" (UID: \"483879f7-8946-4dc9-950f-64f8082d9651\") "
Sep 12 01:23:45.755253 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.755223    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483879f7-8946-4dc9-950f-64f8082d9651-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "483879f7-8946-4dc9-950f-64f8082d9651" (UID: "483879f7-8946-4dc9-950f-64f8082d9651"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:23:45.755383 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.755367    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483879f7-8946-4dc9-950f-64f8082d9651-openshift-service-ca\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:23:45.757191 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.757167    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/483879f7-8946-4dc9-950f-64f8082d9651-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "483879f7-8946-4dc9-950f-64f8082d9651" (UID: "483879f7-8946-4dc9-950f-64f8082d9651"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:23:45.757307 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.757213    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/483879f7-8946-4dc9-950f-64f8082d9651-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "483879f7-8946-4dc9-950f-64f8082d9651" (UID: "483879f7-8946-4dc9-950f-64f8082d9651"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:23:45.757307 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.757251    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/483879f7-8946-4dc9-950f-64f8082d9651-kube-api-access-h6sll" (OuterVolumeSpecName: "kube-api-access-h6sll") pod "483879f7-8946-4dc9-950f-64f8082d9651" (UID: "483879f7-8946-4dc9-950f-64f8082d9651"). InnerVolumeSpecName "kube-api-access-h6sll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:23:45.855885 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.855800    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6sll\" (UniqueName: \"kubernetes.io/projected/483879f7-8946-4dc9-950f-64f8082d9651-kube-api-access-h6sll\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:23:45.855885 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.855828    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483879f7-8946-4dc9-950f-64f8082d9651-maas-api-tls\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:23:45.855885 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.855837    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483879f7-8946-4dc9-950f-64f8082d9651-metrics-certs\") on node \"ip-10-0-139-121.ec2.internal\" DevicePath \"\""
Sep 12 01:23:45.886572 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.886537    2567 generic.go:356] "Generic (PLEG): container finished" podID="483879f7-8946-4dc9-950f-64f8082d9651" containerID="e6496b37c642744a553bd3953cdcb0d5c11135d4a87705d4fd2350a2a745b3f4" exitCode=0
Sep 12 01:23:45.886738 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.886582    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"
Sep 12 01:23:45.886738 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.886589    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl" event={"ID":"483879f7-8946-4dc9-950f-64f8082d9651","Type":"ContainerDied","Data":"e6496b37c642744a553bd3953cdcb0d5c11135d4a87705d4fd2350a2a745b3f4"}
Sep 12 01:23:45.886738 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.886654    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl" event={"ID":"483879f7-8946-4dc9-950f-64f8082d9651","Type":"ContainerDied","Data":"f7b7a3d02b8cfd3c1132da246940f3b7f710a9c70e683e18fe779214b1b0aa25"}
Sep 12 01:23:45.886738 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.886676    2567 scope.go:117] "RemoveContainer" containerID="e6496b37c642744a553bd3953cdcb0d5c11135d4a87705d4fd2350a2a745b3f4"
Sep 12 01:23:45.896318 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.896302    2567 scope.go:117] "RemoveContainer" containerID="e6496b37c642744a553bd3953cdcb0d5c11135d4a87705d4fd2350a2a745b3f4"
Sep 12 01:23:45.896570 ip-10-0-139-121 kubenswrapper[2567]: E0912 01:23:45.896554    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6496b37c642744a553bd3953cdcb0d5c11135d4a87705d4fd2350a2a745b3f4\": container with ID starting with e6496b37c642744a553bd3953cdcb0d5c11135d4a87705d4fd2350a2a745b3f4 not found: ID does not exist" containerID="e6496b37c642744a553bd3953cdcb0d5c11135d4a87705d4fd2350a2a745b3f4"
Sep 12 01:23:45.896621 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.896579    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6496b37c642744a553bd3953cdcb0d5c11135d4a87705d4fd2350a2a745b3f4"} err="failed to get container status \"e6496b37c642744a553bd3953cdcb0d5c11135d4a87705d4fd2350a2a745b3f4\": rpc error: code = NotFound desc = could not find container \"e6496b37c642744a553bd3953cdcb0d5c11135d4a87705d4fd2350a2a745b3f4\": container with ID starting with e6496b37c642744a553bd3953cdcb0d5c11135d4a87705d4fd2350a2a745b3f4 not found: ID does not exist"
Sep 12 01:23:45.910845 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.910818    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"]
Sep 12 01:23:45.914926 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.914904    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc46ff6dc-bwqhl"]
Sep 12 01:23:45.973105 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:23:45.973079    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="483879f7-8946-4dc9-950f-64f8082d9651" path="/var/lib/kubelet/pods/483879f7-8946-4dc9-950f-64f8082d9651/volumes"
Sep 12 01:24:05.442812 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:05.442786    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694bbb96f5-gnqwd_fc585fd3-4864-46bf-801d-7a598e8b8b14/copy-manifests/0.log"
Sep 12 01:24:05.456462 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:05.456437    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694bbb96f5-gnqwd_fc585fd3-4864-46bf-801d-7a598e8b8b14/manager/0.log"
Sep 12 01:24:06.072550 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:06.072525    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-749bf69647-mnx6k_31c1efb3-b874-421b-801b-b6ff16a67180/maas-api/0.log"
Sep 12 01:24:06.539661 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:06.539626    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-t758b_56c87398-ea9e-413d-9d66-e314c3d0e092/postgres/0.log"
Sep 12 01:24:07.705965 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:07.705928    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-m2sqp_6456a19b-fcb4-4955-aee2-ad30ccc7f4ca/kuadrant-console-plugin/0.log"
Sep 12 01:24:08.836758 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:08.836728    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f75b6b767-xz9v9_268927fe-2fda-4277-98ed-679fea20c9ca/kube-auth-proxy/0.log"
Sep 12 01:24:09.086831 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:09.086754    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rzbx5_4579cc0d-433e-486c-a553-d0c76bda8b0d/payload-pre-processing/0.log"
Sep 12 01:24:09.802134 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:09.802101    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-brqqx_ab49529a-5f51-43a6-9314-372c9c54581e/main/0.log"
Sep 12 01:24:10.036140 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:10.036111    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-bhpzd_31a801bd-4fc0-4ea8-ad02-98e22e6c2f79/main/0.log"
Sep 12 01:24:10.372986 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:10.372942    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-6f89h_70d9b75a-7b92-4dbe-afc1-aed86de0ec95/main/0.log"
Sep 12 01:24:16.359367 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:16.359339    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:24:16.459436 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:16.459405    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:24:19.349080 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:19.349052    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5k8lk_d5d08bf0-f998-4798-b16c-ec2c5cecbe21/global-pull-secret-syncer/0.log"
Sep 12 01:24:19.484904 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:19.484844    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log"
Sep 12 01:24:19.610890 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:19.610814    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-121.ec2.internal_ad9778864c1c592078495eb7196d09e3/haproxy/0.log"
Sep 12 01:24:24.793674 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:24.793647    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-m2sqp_6456a19b-fcb4-4955-aee2-ad30ccc7f4ca/kuadrant-console-plugin/0.log"
Sep 12 01:24:26.557577 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:26.557544    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d58fbf47f-fjh87_bc9bb428-0aff-44c1-98a9-4692709adda5/metrics-server/0.log"
Sep 12 01:24:26.838126 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:26.838101    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xcjkz_36a5e6bc-d6e1-49d4-a665-6d7639fd24b2/node-exporter/0.log"
Sep 12 01:24:26.860340 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:26.860319    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xcjkz_36a5e6bc-d6e1-49d4-a665-6d7639fd24b2/kube-rbac-proxy/0.log"
Sep 12 01:24:26.884599 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:26.884575    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xcjkz_36a5e6bc-d6e1-49d4-a665-6d7639fd24b2/init-textfile/0.log"
Sep 12 01:24:27.009959 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:27.009924    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11db4521-b277-45e1-a609-7c48b808b13f/prometheus/0.log"
Sep 12 01:24:27.032348 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:27.032320    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11db4521-b277-45e1-a609-7c48b808b13f/config-reloader/0.log"
Sep 12 01:24:27.056202 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:27.056179    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11db4521-b277-45e1-a609-7c48b808b13f/thanos-sidecar/0.log"
Sep 12 01:24:27.078094 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:27.078041    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11db4521-b277-45e1-a609-7c48b808b13f/kube-rbac-proxy-web/0.log"
Sep 12 01:24:27.102408 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:27.102331    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11db4521-b277-45e1-a609-7c48b808b13f/kube-rbac-proxy/0.log"
Sep 12 01:24:27.125411 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:27.125375    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11db4521-b277-45e1-a609-7c48b808b13f/kube-rbac-proxy-thanos/0.log"
Sep 12 01:24:27.150493 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:27.150476    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11db4521-b277-45e1-a609-7c48b808b13f/init-config-reloader/0.log"
Sep 12 01:24:28.998154 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:28.998129    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-svchm_653056a7-146e-44bd-bd1d-41cd75537411/networking-console-plugin/0.log"
Sep 12 01:24:29.523254 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.523224    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"]
Sep 12 01:24:29.523587 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.523575    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483879f7-8946-4dc9-950f-64f8082d9651" containerName="maas-api"
Sep 12 01:24:29.523631 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.523589    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="483879f7-8946-4dc9-950f-64f8082d9651" containerName="maas-api"
Sep 12 01:24:29.523667 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.523651    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="483879f7-8946-4dc9-950f-64f8082d9651" containerName="maas-api"
Sep 12 01:24:29.526899 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.526872    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.529469 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.529447    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgbmv\"/\"openshift-service-ca.crt\""
Sep 12 01:24:29.530640 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.530622    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hgbmv\"/\"default-dockercfg-dwdhd\""
Sep 12 01:24:29.530815 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.530644    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgbmv\"/\"kube-root-ca.crt\""
Sep 12 01:24:29.536066 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.536045    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"]
Sep 12 01:24:29.552062 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.552029    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65d92323-0a29-43c4-bfee-9ce868a53687-lib-modules\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.552193 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.552077    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65d92323-0a29-43c4-bfee-9ce868a53687-proc\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.552193 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.552164    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65d92323-0a29-43c4-bfee-9ce868a53687-sys\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.552193 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.552181    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65d92323-0a29-43c4-bfee-9ce868a53687-podres\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.552337 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.552205    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md7lf\" (UniqueName: \"kubernetes.io/projected/65d92323-0a29-43c4-bfee-9ce868a53687-kube-api-access-md7lf\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.653313 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.653276    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65d92323-0a29-43c4-bfee-9ce868a53687-sys\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.653313 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.653313    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65d92323-0a29-43c4-bfee-9ce868a53687-podres\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.653621 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.653332    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md7lf\" (UniqueName: \"kubernetes.io/projected/65d92323-0a29-43c4-bfee-9ce868a53687-kube-api-access-md7lf\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.653621 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.653367    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65d92323-0a29-43c4-bfee-9ce868a53687-lib-modules\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.653621 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.653392    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65d92323-0a29-43c4-bfee-9ce868a53687-proc\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.653621 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.653396    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65d92323-0a29-43c4-bfee-9ce868a53687-sys\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.653621 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.653449    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65d92323-0a29-43c4-bfee-9ce868a53687-proc\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.653621 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.653464    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65d92323-0a29-43c4-bfee-9ce868a53687-podres\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.653621 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.653565    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65d92323-0a29-43c4-bfee-9ce868a53687-lib-modules\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.656135 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.656114    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/1.log"
Sep 12 01:24:29.660139 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.660119    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gbjdw_09725ba5-1c2a-4a09-9901-65dcf94d47d4/console-operator/2.log"
Sep 12 01:24:29.663730 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.663702    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md7lf\" (UniqueName: \"kubernetes.io/projected/65d92323-0a29-43c4-bfee-9ce868a53687-kube-api-access-md7lf\") pod \"perf-node-gather-daemonset-7wppg\" (UID: \"65d92323-0a29-43c4-bfee-9ce868a53687\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:29.837446 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:29.837349    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:30.109352 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:30.109326    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"]
Sep 12 01:24:30.111707 ip-10-0-139-121 kubenswrapper[2567]: W0912 01:24:30.111679    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod65d92323_0a29_43c4_bfee_9ce868a53687.slice/crio-2d6b6e17a75f9ed6028a8340bd4976b6fa3c003dc041e30f7509560ac4e9082b WatchSource:0}: Error finding container 2d6b6e17a75f9ed6028a8340bd4976b6fa3c003dc041e30f7509560ac4e9082b: Status 404 returned error can't find the container with id 2d6b6e17a75f9ed6028a8340bd4976b6fa3c003dc041e30f7509560ac4e9082b
Sep 12 01:24:30.311176 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:30.311147    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-4s92h_e8b9fcfb-2862-479f-b158-7c3c3f4ab57f/download-server/0.log"
Sep 12 01:24:31.073176 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:31.073152    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-v9xsz_08374dad-0ebd-4626-8470-566aa010ba48/volume-data-source-validator/0.log"
Sep 12 01:24:31.102830 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:31.102797    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg" event={"ID":"65d92323-0a29-43c4-bfee-9ce868a53687","Type":"ContainerStarted","Data":"76b081109060fccf1d4aa666b5328a913a3985793ffb40870a87962fcf220aa2"}
Sep 12 01:24:31.103003 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:31.102855    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg" event={"ID":"65d92323-0a29-43c4-bfee-9ce868a53687","Type":"ContainerStarted","Data":"2d6b6e17a75f9ed6028a8340bd4976b6fa3c003dc041e30f7509560ac4e9082b"}
Sep 12 01:24:31.103003 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:31.102885    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:31.121844 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:31.121800    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg" podStartSLOduration=2.121784384 podStartE2EDuration="2.121784384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:24:31.120035902 +0000 UTC m=+2475.680487598" watchObservedRunningTime="2026-09-12 01:24:31.121784384 +0000 UTC m=+2475.682236053"
Sep 12 01:24:32.190507 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:32.190477    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dsbc2_d07d13ab-1c44-499a-aa9f-ebed6bbe188e/dns/0.log"
Sep 12 01:24:32.213534 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:32.213506    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dsbc2_d07d13ab-1c44-499a-aa9f-ebed6bbe188e/kube-rbac-proxy/0.log"
Sep 12 01:24:32.312223 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:32.312198    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cbjgx_2337c617-d3c5-417f-b2c8-4c8328a04a58/dns-node-resolver/0.log"
Sep 12 01:24:32.883470 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:32.883437    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-58cf8758-nm4j8_e894c61f-cb95-48ed-a6dc-a2d77b165617/registry/0.log"
Sep 12 01:24:33.001321 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:33.001288    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mvtjp_5bf0d680-e19c-4892-8652-58f49ed9d3b3/node-ca/0.log"
Sep 12 01:24:34.822158 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:34.822126    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f75b6b767-xz9v9_268927fe-2fda-4277-98ed-679fea20c9ca/kube-auth-proxy/0.log"
Sep 12 01:24:34.911715 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:34.911681    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rzbx5_4579cc0d-433e-486c-a553-d0c76bda8b0d/payload-pre-processing/0.log"
Sep 12 01:24:35.900301 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:35.900271    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6z6bf_2ee67b56-1998-4545-9ceb-8597c6101dab/serve-healthcheck-canary/0.log"
Sep 12 01:24:36.387140 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:36.387102    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-26rk9_aec6dfd1-8c25-4904-9234-23f02c32ea46/insights-operator/0.log"
Sep 12 01:24:36.388573 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:36.388556    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-26rk9_aec6dfd1-8c25-4904-9234-23f02c32ea46/insights-operator/1.log"
Sep 12 01:24:36.484147 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:36.484124    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rp657_19b79c5c-195d-436e-8785-90cc7e91d8cb/kube-rbac-proxy/0.log"
Sep 12 01:24:36.508710 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:36.508687    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rp657_19b79c5c-195d-436e-8785-90cc7e91d8cb/exporter/0.log"
Sep 12 01:24:36.531020 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:36.530997    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rp657_19b79c5c-195d-436e-8785-90cc7e91d8cb/extractor/0.log"
Sep 12 01:24:37.118545 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:37.118519    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-7wppg"
Sep 12 01:24:38.154737 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:38.154706    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5brn_cf7cbf7f-c916-4a10-a6f2-4e562a1edc5c/konnectivity-agent/0.log" limit=1024
Sep 12 01:24:39.628227 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:39.628155    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694bbb96f5-gnqwd_fc585fd3-4864-46bf-801d-7a598e8b8b14/manager/0.log"
Sep 12 01:24:39.646056 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:39.646032    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694bbb96f5-gnqwd_fc585fd3-4864-46bf-801d-7a598e8b8b14/copy-manifests/0.log"
Sep 12 01:24:40.926486 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:40.926456    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6ccd6666c9-vzvpv_c090878f-bc8f-40a3-af92-acce749fe642/manager/0.log"
Sep 12 01:24:41.136972 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:41.136925    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f75b6b767-xz9v9_268927fe-2fda-4277-98ed-679fea20c9ca/kube-auth-proxy/0.log"
Sep 12 01:24:41.181644 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:41.181617    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rzbx5_4579cc0d-433e-486c-a553-d0c76bda8b0d/payload-pre-processing/0.log"
Sep 12 01:24:43.542590 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:43.542561    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f75b6b767-xz9v9_268927fe-2fda-4277-98ed-679fea20c9ca/kube-auth-proxy/0.log"
Sep 12 01:24:43.594383 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:43.594352    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rzbx5_4579cc0d-433e-486c-a553-d0c76bda8b0d/payload-pre-processing/0.log"
Sep 12 01:24:45.923153 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:45.923124    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-hdt7r_24e688c4-27f7-4830-969b-af15d8962683/migrator/0.log"
Sep 12 01:24:45.946324 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:45.946290    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-hdt7r_24e688c4-27f7-4830-969b-af15d8962683/graceful-termination/0.log"
Sep 12 01:24:46.295839 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:46.295763    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-5pnzt_a58e1e17-be8b-47d5-bdee-cd38e5554c44/kube-storage-version-migrator-operator/1.log"
Sep 12 01:24:46.296587 ip-10-0-139-121 kubenswrapper[2567]: I0912 01:24:46.296569    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-5pnzt_a58e1e17-be8b-47d5-bdee-cd38e5554c44/kube-storage-version-migrator-operator/0.log"